{"id":"a7495301-44f4-43af-90e5-b29767ad9bd6","arxiv_id":"1908.06232","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":11,"one_line_summary":"NSGA-II, SPEA-II and MOEA/D, embedded in a multi-objective NARX structure-selection framework, recover correct structures on benchmarks, with NSGA-II ranked best and wide parameter sweet spots.","lead":"This paper builds a multi-objective evolutionary framework, MOSS, that selects compact NARX models by trading off model size against prediction error. It tests three well-known MOEAs on eight benchmark systems and shows they can recover known structures and produce multiple valid discrete-time models for continuous systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GFRF comparison in Section 5.3 is qualitative and truncated at third order, so the multiple-valid-models conclusion for the Duffing oscillator is not established with certainty.","rationale":"The reader's weakest assumption identifies exactly this GFRF truncation issue, and my reading agrees. The benchmark recovery in Table 2 and the statistical comparison of MOEAs are largely self-contained and empirically convincing; the main soft spot is the continuous-time case study where GFRF agreement is used as a validity certificate. The concern does not by itself refute the paper, because matching finite-order GFRFs is meaningful evidence, but it is not conclusive, and the paper's strongest claim overstates the strength of the evidence. A CONDITIONAL verdict is appropriate: the multiple-valid-models claim should be either quantitatively validated at higher order or explicitly softened to a claim of agreement up to the third-order GFRFs.","tokens_in":31383,"tokens_out":1281,"duration_ms":15108,"concrete_test":"Compute the fourth-order GFRFs of MD1, MD2, and MD3 by harmonic probing of the identified NARX models, evaluate them on the same frequency grid used for the continuous-time Duffing oscillator, and compute a normalized relative error between each model's H4 and the continuous-time H4. If the errors are nonzero and not within a small tolerance (e.g., <5%), the finite-order GFRF match does not establish model validity, and the Section 5.3 claim should be softened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that MOSS can identify multiple valid discrete-time models for continuous-time systems rests on the Duffing case study in Section 5.3. The evidence is a visual comparison of linear and third-order GFRFs between MD1-MD3 and the continuous-time oscillator. No quantitative error or tolerance is reported, and only first- and third-order kernels are compared. Since the Volterra series of a nonlinear system continues beyond third order, and two distinct polynomial NARX models can match in low-order kernels while differing at higher order, matching H1 and H3 does not, by itself, establish that MD1-MD3 are valid models of the Duffing dynamics. The paper asserts that GFRFs are invariant descriptors, but it does not provide a theorem or derivation showing that agreement of finitely many low-order GFRFs implies dynamic equivalence. This is load-bearing because the paper's headline contribution, i.e., multiple valid discrete models, depends on this validation being conclusive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Multi-Objective Structure Selection (MOSS) framework for polynomial NARX model identification, combining preference articulation, three MOEAs (NSGA-II, SPEA-II, MOEA/D), and a posteriori selection via MMD or MTD. The framework is evaluated on seven discrete-time benchmark systems, a Duffing oscillator case study, and a nonlinear wave-force identification case study. The main claims are that all compared MOEAs can identify the correct structure on the discrete benchmarks, that NSGA-II is significantly better than SPEA-II and MOEA/D in terms of hypervolume, that the algorithms are robust over large 'sweet-spot' regions in parameter space, and that frequency-domain GFRF analysis establishes the existence of multiple valid discrete-time models for continuous-time systems.","tokens_in":31606,"tokens_out":4353,"duration_ms":46008,"significance":"If the claims are fully substantiated, the paper would be a useful contribution: the MOSS formulation with a two-objective (cardinality, NMSE) problem, explicit goal-based preference articulation, and systematic MOEA comparison is sensible and fills a gap in the NARX structure-selection literature. The benchmark recovery results in Table 2 are clean and the sweet-spot analysis in Section 7 is a valuable practical addition, especially because the paper reports full parameter settings and identified model equations, which aids reproducibility. The main limitations are that the central 'multiple valid models' conclusion rests on a qualitative and truncated GFRF comparison, and the statistical comparison of MOEAs does not report run-to-run variability; both weaken the strength of the claimed conclusions.","major_comments":[{"comment":"The conclusion that MD1-MD3 are conclusively valid discrete-time models of the Duffing oscillator is supported only by visual agreement of the first- and third-order GFRFs with the continuous-time model. The paper asserts that GFRFs are invariant descriptors, but it does not provide a theorem or quantitative bound showing that matching finitely many low-order kernels implies dynamic equivalence. Since a polynomial NARX model has Volterra kernels of all orders, two distinct models can agree on H1 and H3 while differing at higher orders. To support the central claim, the authors should report quantitative kernel errors (e.g., normalized L2 or max deviation over the plotted frequency ranges), compare at least one higher-order kernel or a nonlinear-response prediction test, or use an independent validation signal; as it stands, the evidence is suggestive rather than conclusive.","section":"Section 5.3, Eqs. (30)-(32), Figs. 4-6"},{"comment":"The statistical claim that NSGA-II is significantly better than SPEA-II and MOEA/D is based on a single hypervolume value per algorithm per test system, obtained from the pooled approximate Pareto set of 40 runs. No run-to-run variance of the HV indicator is reported, so the Friedman test effectively has only nine blocks and does not account for the stochasticity of the algorithms. The authors should report the distribution of HV over independent runs and perform the test on per-run values, or carefully justify why pooling is valid for significance testing. In addition, Table 7 is internally unclear: SPEA-II is reported with p=0.0593 but an adjusted p-value of 0.050 and an apparently rejected null hypothesis; the test statistic, adjusted p-values, and decision rule need to be stated consistently.","section":"Section 6.2, Tables 6-7"},{"comment":"The crossover and mutation probabilities for each MOEA were selected from the sweet-spot analysis in Section 7.2, which was performed on the same benchmark systems (S1-S7) that are later used for the comparative evaluation in Sections 5 and 6. This is an in-sample parameter selection, so the reported comparisons and the robustness conclusion are not fully independent. The authors should either perform an out-of-sample or cross-validated parameter selection, or explicitly quantify how the conclusions depend on the selected parameters; at minimum, this limitation should be acknowledged in the text.","section":"Section 4.1 and Section 7.2"}],"minor_comments":[{"comment":"The row and column alignment in Table 2 appears corrupted in the typeset version, especially for the 'Under-fitting' rows; the counts should be re-formatted so that each column clearly corresponds to NSGA-II, SPEA-II, and MOEA/D.","section":"Table 2"},{"comment":"The GFRF figures lack color-bar scales and axis labels on several panels, making a quantitative reading difficult; adding color bars and consistent frequency grid labels would strengthen the qualitative comparison.","section":"Section 5.3, Figs. 4-6"},{"comment":"Several hypervolume values are identical across algorithms to four decimal places (e.g., S2 and the Duffing oscillator); the authors should report more decimal places or indicate the actual numerical differences so that ties are not merely artifacts of rounding.","section":"Section 6, Table 6"},{"comment":"The statement that all MOEAs have been successful in identifying the correct structure is based on structures that are first refined by a null-hypothesis test on the coefficients; this should be stated in Section 5.1's concluding sentence to avoid overstating the raw search outcome.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core ideas are sound and the empirical material is substantial, but the two load-bearing points—the GFRF-based validation of multiple valid discrete models and the statistical comparison of MOEAs—need strengthening before publication. The in-sample parameter tuning is a further concern that should be addressed or explicitly acknowledged. I do not see a novelty or scope problem; the manuscript fits the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a careful, honest empirical study. It does not pretend multi-objective structure selection is new—it credits Fonseca et al. from the 1990s—and its contribution is a systematic comparison of NSGA-II, SPEA-II and MOEA/D on NARX structure selection, with preference articulation, a posteriori MMD/MTD ranking, sweet-spot parameter maps, and frequency-domain validation on a Duffing oscillator and a wave-force dataset. The benchmark recovery results are clean: on the six discrete-time systems, every MOEA finds at least one exact-fitting structure in every case, and Table 2 is informative.\n\nThe paper does several things well. The framework is described in enough detail to reproduce (population size, function evaluations, crossover operators, goal {20, 30} penalty). The sweet-spot analysis is a sensible way to characterize robustness, and the Wilcoxon comparison of crossover mechanisms is appropriate. The statistical comparison via Friedman + Hommel on hypervolume is standard, though see below.\n\nSoft spots, in order of importance:\n\n1. The GFRF validation is the weakest part. The paper claims the identified models MD1–MD3 give \"conclusive evidence\" that they describe the Duffing dynamics, based only on visual matching of first- and third-order GFRFs. No quantitative error or tolerance is given, and no argument that matching H1 and H3 implies dynamic equivalence. Two distinct polynomial NARX models can share low-order kernels and differ at higher orders. This does not break the framework, but the claim should be softened to \"consistent with the system's dominant frequency-domain behavior\" rather than \"conclusive\".\n\n2. Parameter tuning circularity is real but minor. The sweet-spot analysis selects {pc, pm} for each algorithm, and those same settings are used in the reported comparisons and the statistical test. That is common in empirical MOEA papers, but the paper should acknowledge that the comparisons are conditional on the chosen settings.\n\n3. The hypervolume-based Friedman test uses one HV value per algorithm per system, with no run-to-run variance. The \"significantly better\" claim therefore rests on a single number per cell; boxplots across the 40 runs would be more convincing. The result is plausible, but the evidence is thinner than the prose suggests.\n\n4. The robustness conclusion is qualitative; the contour maps show large sweet spots, but \"large\" is not quantified.\n\nThe math and citation pattern look fine. The paper cites the relevant prior MOEA and NARX literature, including the original Fonseca work, and does not oversell its novelty.\n\nWho this is for: anyone working on evolutionary structure selection or NARX identification will get practical value. It deserves a serious referee; I would suggest major revision, not rejection—the GFRF claim needs reining in and the statistical presentation needs variance information. If those are addressed, I would be happy to see it in print.","headline":"A solid, reproducible empirical comparison of three MOEAs for NARX structure selection; the multi-model GFRF validation claim is overstated, but the central framework conclusion holds.","tokens_in":32167,"tokens_out":2971,"would_cite":true,"duration_ms":31149,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a two-objective evolutionary framework that selects NARX model structures and reports that it recovers correct structures for all tested benchmark systems.","keywords":["nonlinear system identification","NARX model","structure selection","multi-objective evolutionary algorithms","NSGA-II","SPEA-II","MOEA/D","Generalized Frequency Response Functions"],"falsifier":"Compute the fifth-order Generalized Frequency Response Functions of the identified models MD1, MD2 and MD3 by the same harmonic-probing recursion and compare them with the continuous-time Duffing oscillator's fifth-order kernel; if the higher-order kernels diverge substantially while the first- and third-order kernels agree, then matching only the low-order responses is not enough to prove the models valid.","tokens_in":31157,"feed_emoji":"🎯","tokens_out":4026,"duration_ms":38413,"temperature":0.7,"pith_summary":"The paper proposes a multi-objective framework for choosing the structure of polynomial NARX models of nonlinear systems, framing structure selection as a two-objective problem that minimizes prediction error and model cardinality. It claims that, with a posteriori decision rules and preference-based goals, standard multi-objective evolutionary algorithms reliably recover the correct structure of benchmark discrete-time nonlinear systems, and that NSGA-II is statistically significantly better than SPEA-II and MOEA/D for the search. It also argues that the framework's key advantage is producing several valid discrete-time models for a continuous-time system, validated by matching Generalized Frequency Response Functions up to third order. A sympathetic reader would care because structure selection is the bottleneck in nonlinear system identification, and the paper offers a practical workflow plus statistical evidence about which optimizer to use.","feed_headline":"NSGA-II tops two rivals on nonlinear structure selection","feed_subtitle":"A two-objective search recovers true NARX terms and yields several valid discrete models for continuous-time systems.","key_machinery":"The load-bearing object is the Approximate Pareto Set of non-dominated NARX structures, obtained by minimizing the pair $\\{\\text{cardinality }\\xi,\\ \\text{normalized mean-squared error }E\\}$. The machinery combines a binary encoding of candidate term subsets; three multi-objective evolutionary algorithms, NSGA-II, SPEA-II and MOEA/D, that search the space; a goal-point penalty that focuses the search at $\\xi\\le 20$ and $E\\le 30$; and the a posteriori ranking rules Minimum Manhattan Distance (MMD) and Multi-criteria Tournament Decision (MTD) that pick final models. The identity doing the validation work is the Generalized Frequency Response Function, computed from identified models by harmonic probing; the paper relies on these functions as invariant descriptors of the system dynamics that should match whenever two models describe the same system.","core_discovery":"The central claim is that structure selection for polynomial NARX models can be solved as a two-objective optimization problem—minimize normalized mean-squared prediction error $E$ and number of terms $\\xi$—and that the non-dominated set of structures, generated by NSGA-II, SPEA-II or MOEA/D with a goal point $\\{\\xi\\le 20,\\ E\\le 30\\}$ and ranked by Minimum Manhattan Distance or Multi-criteria Tournament Decision, contains the correct structure for every benchmark discrete-time system tested. For continuous-time systems, the framework discovers multiple distinct discrete-time NARX models, and the paper asserts these models are all valid because their first- and third-order Generalized Frequency Response Functions match those of the original system. On the search side, the paper claims NSGA-II is significantly better than SPEA-II and MOEA/D (Friedman test $p=0.0084$, with Hommel-adjusted post-hoc rejection of both null hypotheses), and that the algorithms are robust across a wide sweet spot of crossover and mutation probabilities.","pith_inferences":["If matching low-order Generalized Frequency Response Functions is taken as a general validity test, the same framework could be extended to compare fourth- and fifth-order kernels, or to use frequency-domain coherence, as a stricter check for systems whose discrete representation is not unique.","The multiple-valid-models result suggests a natural ensemble-identification extension: instead of selecting one model, average predictions of the top-ranked non-dominated structures, which the paper does not explore.","The sensitivity analysis leaves MOEA/D's own parameters, such as neighborhood size, replacement pool size, and aggregation function, untested; those are plausible drivers of its weaker robustness, not just the crossover choice.","The goal point $\\{\\xi\\le 20,\\ E\\le 30\\}$ needs updating when the true system has more than 20 terms, and the paper's Section 5.2 shows how to detect this via information criteria, so the preference-update step could be automated."],"forward_implications":["If the workflow is applied to a new nonlinear system, the user can expect an approximate Pareto set whose top-ranked structures by MMD or MTD include exact or near-exact structures for discrete-time systems.","NSGA-II with uniform crossover and crossover/mutation probabilities around $p_c=0.9$, $p_m=0.006$ is a reasonable default, since it outperforms the other two algorithms on six of nine test systems.","For continuous-time systems, multiple distinct discrete-time NARX models can be regarded as valid as long as their low-order Generalized Frequency Response Functions match, so parsimony can guide the final choice.","The identified performance sweet spots in the parameter space can serve as rule-of-thumb starting points for tuning the same algorithms on other identification problems.","Uniform crossover is preferable to single-point crossover for these MOEAs, and for MOEA/D the improvement is statistically significant on every test system."],"supporting_citations":[{"why":"Supplies the NSGA-II algorithm, one of the three multi-objective optimizers compared in the framework.","marker":"[38]"},{"why":"Supplies the SPEA-II algorithm with its archive-based elitism and density estimation.","marker":"[39]"},{"why":"Supplies the MOEA/D decomposition-based algorithm used as the third optimizer.","marker":"[40]"},{"why":"Provides the harmonic-probing procedure used to compute Generalized Frequency Response Functions from the identified NARX models.","marker":"[12]"},{"why":"Establishes the NARX representation, the non-uniqueness of discrete models for continuous-time systems, and the frequency-domain tools that justify GFRF validation.","marker":"[4]"},{"why":"Defines the Minimum Manhattan Distance approach used for a posteriori selection of final structures.","marker":"[67]"},{"why":"Defines the Multi-criteria Tournament Decision method used when the decision maker supplies preference weights.","marker":"[66]"},{"why":"Supplies the practical nonlinear wave-force dataset used in the case study.","marker":"[76]"},{"why":"Provides the Friedman test and Hommel post-hoc procedure used to establish the statistical significance of the NSGA-II advantage.","marker":"[77]"}],"fun_headline_variants":["NSGA-II sweeps SPEA-II and MOEA/D on NARX structure selection","Two-objective evolution hits true NARX terms every benchmark","Continuous-time systems give multiple valid discrete models","MOEA parameter sweet spots ensure robust structure search","Multi-objective search nails nonlinear system identification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats matching of first- and third-order Generalized Frequency Response Functions between an identified discrete-time model and the original continuous-time system as sufficient evidence that the model is dynamically valid.","fun_headline_variants_meta":{"raw":{"variants":["NSGA-II sweeps SPEA-II and MOEA/D on NARX structure selection","Two-objective evolution hits true NARX terms every benchmark","Continuous-time systems give multiple valid discrete models","MOEA parameter sweet spots ensure robust structure search","Multi-objective search nails nonlinear system identification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1803,"prompt_tokens":970,"completion_tokens":833,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":752}},"tokens_in":586,"tokens_out":833,"duration_ms":8279,"temperature":1.0,"reasoning_tokens":752,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:52:20.500540+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the fifth-order Generalized Frequency Response Functions of the identified models MD1, MD2 and MD3 by the same harmonic-probing recursion and compare them with the continuous-time Duffing oscillator's fifth-order kernel; if the higher-order kernels diverge substantially while the first- and third-order kernels agree, then matching only the low-order responses is not enough to prove the models valid.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NSGA-II algorithm, one of the three multi-objective optimizers compared in the framework."},{"cited_title":"Zitzler, M","cited_arxiv_id":null,"evidence_quote":"Supplies the SPEA-II algorithm with its archive-based elitism and density estimation."},{"cited_title":"Zhang, H","cited_arxiv_id":null,"evidence_quote":"Supplies the MOEA/D decomposition-based algorithm used as the third optimizer."},{"cited_title":"Billings, K","cited_arxiv_id":null,"evidence_quote":"Provides the harmonic-probing procedure used to compute Generalized Frequency Response Functions from the identified NARX models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the NARX representation, the non-uniqueness of discrete models for continuous-time systems, and the frequency-domain tools that justify GFRF validation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Minimum Manhattan Distance approach used for a posteriori selection of final structures."},{"cited_title":"Parreiras, J","cited_arxiv_id":null,"evidence_quote":"Defines the Multi-criteria Tournament Decision method used when the decision maker supplies preference weights."},{"cited_title":"Swain, S","cited_arxiv_id":null,"evidence_quote":"Supplies the practical nonlinear wave-force dataset used in the case study."},{"cited_title":"Derrac, S","cited_arxiv_id":null,"evidence_quote":"Provides the Friedman test and Hommel post-hoc procedure used to establish the statistical significance of the NSGA-II advantage."}],"review_version":1}