{"id":"95b8bed1-085e-43cb-ae2d-aa9fe50df395","arxiv_id":"2411.19387","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ES-NEAT calibrates about 20,000 parameters of a drinking water network model to an RMSE of 0.33 (validation 0.56) using expert-system bounds and evolved neural networks.","lead":"This paper introduces ES-NEAT, a hybrid method that combines expert-system rules with neuroevolution to calibrate computer models of drinking water distribution networks. It reports accurate fits on a real Belgian network of about 20,000 parameters using limited measurements and moderate computation, plus a mechanism to carry calibration knowledge into future calibrations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Identifiability of ~20,000 calibrated parameters from sparse measurements (42 nodes, 8 pipes) is not established; low RMSE may reflect compensation rather than correct global calibration.","rationale":"The reader's weakest assumption is exactly the identifiability of the high-dimensional parameter vector from sparse measurements, and I agree with it. This is the most load-bearing concern because the paper's headline contribution is 'global calibration of parameters' from limited data; if the parameters are not identifiable, the RMSE numbers only show that ES-NEAT can fit the training sensors, not that it has found correct network-wide parameters. The paper includes no parameter-recovery experiment, no sensitivity or identifiability analysis, and no comparison of solutions from repeated runs, and the validation set is too small. These omissions are particularly important because the same limitation is acknowledged in the Introduction, so the authors are aware of the ill-posedness but do not show how ES-NEAT or the expert-system priors overcome it. I still regard the current verdict as CONDITIONAL rather than REJECT: the case study is real, the before/after fits are visually substantial, and the method may well work in practice; the missing evidence is obtainable. The proposed synthetic recovery test would directly settle the matter, and the lack of code or data release makes independent verification harder, but that is a reproducibility issue rather than the core scientific objection.","tokens_in":13493,"tokens_out":4454,"duration_ms":38400,"concrete_test":"Using the real network model, take the ES-NEAT solution as the 'true' parameter vector, simulate the same one-week average pressure/flow at the same 42 nodes and 8 pipes, add realistic sensor noise, and re-run ES-NEAT from at least five independent initializations. Then compare each recovered parameter vector with the truth. If flow/pressure RMSE values comparable to 0.33 are achieved while the median absolute percentage error across the ~20,000 parameters remains large (e.g., >50%), the parameters are not identifiable from these measurements and the headline claim of global calibration is unsupported. If the recovered parameters match the truth within an acceptable tolerance, the identifiability concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the sparse field data can identify the ~20,000 calibrated parameters. Section 2.1.2 describes measurements at 42 nodes and 8 pipes (with 3 nodes and 2 pipes used for validation) out of 5,197 nodes and 4,158 pipes; Section 2.3 lists the parameters to be calibrated (base demands, leaks, roughnesses, minor losses); Section 3.2 reports calibrating around 20,000 parameters. The paper itself states in the Introduction that the number of parameters typically outnumbers measurements, making DWDN calibration an ill-posed problem, but it never analyzes identifiability for this case. The expert-system rules restrict parameter ranges, but many combinations of demand, leak, and roughness values can still produce nearly identical pressures and flows at the measured locations. Therefore the low calibration RMSE (0.33) and validation RMSE (0.56) do not by themselves establish that the recovered parameters are correct; the calibration could be fitting the sensors through compensating parameter errors. The validation set (3 nodes, 2 pipes) is too small to distinguish a globally calibrated model from one that overfits the calibration sensors.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ES-NEAT, a calibration framework for drinking water distribution network (DWDN) models that combines expert systems (ES) with neuro-evolution of augmenting topologies (NEAT). The method is first benchmarked against several existing optimization algorithms on a small test network, and then applied to a real DWDN in Belgium with roughly 20,000 calibrated parameters, reporting a calibration RMSE of 0.33, a validation RMSE of 0.56, and a computational cost of about 10,000 EPANET simulations (12 hours). The paper also claims that the trained ANN stores calibration information that can be transferred to accelerate future calibration efforts.","tokens_in":13676,"tokens_out":4000,"duration_ms":34189,"significance":"If the reported results are reproducible and generalizable, ES-NEAT would be a practically valuable contribution: it addresses the high-dimensional, underdetermined calibration problem with a moderate computational budget, and the explicit goal of reusing calibration knowledge is a real need in DWDN practice. The benchmark comparison across many optimization methods is a useful resource, and the real-network demonstration is a positive step beyond purely synthetic tests. However, the significance is currently limited because the identifiability of the 20,000-parameter calibration is not analyzed, the real-network evaluation lacks comparison against alternative automated methods on the same data, and the transfer-learning advantage is asserted without a controlled experiment. These gaps prevent the paper from being a definitive demonstration of the method's claimed superiority.","major_comments":[{"comment":"The manuscript calibrates approximately 20,000 parameters (base demands, leaks, roughnesses, minor losses) from only 42 pressure nodes and 8 flow pipes, with 3 nodes and 2 pipes held out for validation. The Introduction itself states that the number of parameters typically outnumbers measurements, making the problem ill-posed, yet no identifiability, sensitivity, or uncertainty analysis is provided. The low RMSE values therefore do not establish that the recovered parameters are physically correct rather than a compensating combination of errors that matches the calibration sensors. The validation RMSE of 0.56 is computed on only 5 locations, which is too few to support the claim of 'global calibration.' Please add an identifiability analysis (e.g., Jacobian rank, parameter-uncertainty bounds, or a synthetic-recovery test) and justify the adequacy of the validation set.","section":"Sections 2.1.2, 2.3, 3.2"},{"comment":"The real-network evaluation compares ES-NEAT only against the prior manual calibration and against two literature studies on different networks (Meirelles et al., 2017; Nicolini et al., 2010). Because the benchmark-network ranking in Section 3.1 may not transfer to the real network's scale, data sparsity, and parameterization, the claim that ES-NEAT is the best automated method is not established. Please run at least one alternative automated calibration method (e.g., SCE-UA or PSO) on the same real network with the same measurement data and computational budget, and report comparative RMSE and convergence behavior.","section":"Section 3.2"},{"comment":"The transfer-learning capability is a stated objective of the paper (Introduction, aim iii), but the only support is a conceptual description and an illustrative figure. No experiment shows that a subsequent calibration initialized from the stored ANN reaches the same accuracy faster or with fewer simulations than a cold start. Please provide a controlled warm-start versus cold-start experiment on either the benchmark or the real network, reporting convergence curves, final RMSE, and total simulation counts.","section":"Section 3.3"},{"comment":"The expert-system rules are described qualitatively (Figure 4) and the NEAT hyperparameters are listed in Section 3.2, but the actual rule base, parameter bounds, the ES inference procedure, and the exact mapping of network attributes to ANN inputs are not specified. No code, data, or network files are provided. This makes the method irreproducible as a methodological contribution. Please make the implementation artifacts available or provide a complete pseudocode, a full rule table, and the calibrated parameter-range tables so that an independent implementation is possible.","section":"Sections 2.3 and 2.4"}],"minor_comments":[{"comment":"The second figure in this section is captioned 'Fig. 1' but is referred to in the text as Figure 8; renumber the figure.","section":"Section 3.3"},{"comment":"The caption contains a duplicated '(a)' in 'The layout of the real network (a)'; the label should be cleaned up.","section":"Figure 2 caption"},{"comment":"The statement that 'a value of less than 0.6 is considered good' and 'accepted by the community' lacks a citation; please provide a reference or justify this threshold.","section":"Section 3.2"},{"comment":"The benchmark comparison reports single-run RMSE values for each method, but the stochastic algorithms (e.g., GA, PSO, NEAT) should be run with multiple random seeds to assess variability and statistical significance of the ranking.","section":"Section 3.1"},{"comment":"The text refers to 'the work developed in the Ostrich automatic calibration tool software' but the citation (Shahed Behrouz et al., 2020) describes a SWMM calibration tool; clarify the connection to Ostrich.","section":"Section 2.2"},{"comment":"Several references are incomplete or inconsistent (e.g., 'Jain and Singh (2003)' appears without initials in the text, and some entries have irregular spacing); please standardize the reference formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Calibrating a real DWDN with about 20,000 parameters from 42 pressure/flow nodes and 8 pipes, and getting a validation RMSE of 0.56, is the kind of result that warrants attention. ES-NEAT is a genuine novelty: the combination of expert-system constraints with NEAT for this problem has not appeared before. The paper builds on the first author's thesis, which already applied NEAT to DWDN calibration, and the authors say so. What is actually new is the ES integration and the explicit goal of storing calibration information for reuse.\n\nWhat the paper does well: it runs a head-to-head comparison of sixteen optimization methods on a benchmark network under a common iteration budget, and the case study is a real network with an actual utility. The before/after plots are convincing. Calibration is evaluated on held-out nodes and pipes, and the validation RMSE is reported — more than many calibration papers do. The paper also states plainly that DWDN calibration is ill-posed, which is honest.\n\nThe soft spots are real but not fatal. The identifiability concern is load-bearing: with 42 nodes and 8 pipes out of over 5,000 nodes and 4,000 pipes, and one year of data compressed to an average week, it is not established that the ~20,000 base demands, leaks, roughnesses, and minor losses are identifiable. Expert-system bounds restrict the search space but do not by themselves solve the problem. The low RMSE could come from compensating parameter errors, and a validation set of 3 nodes and 2 pipes is too small to rule out overfitting. The paper needs either a synthetic test with known true parameters, an identifiability analysis, or at least multiple restarts to show stability. None is provided. The transfer-learning claim is asserted but not tested: there is no experiment showing that initializing from the stored ANN speeds up a subsequent calibration. The comparison with prior methods is narrative, not a same-network control. And no code, data, or network files are provided, so the expert-system rules cannot be inspected or reproduced.\n\nThis is an applied paper for the hydraulic-modeling community. It is not going to rearrange the field, but the problem it tackles matters and the method is plausible. I would send it to peer review with a request for major revisions: supply the artifacts, add an identifiability or synthetic-validation experiment, and actually test the transfer-learning claim. The right referee is someone who understands both hydraulic inverse problems and metaheuristics.\n\nFor a reading group, it would make a good one-session discussion, mostly about why identifiability is the first question to ask when you see a small calibration set and a huge parameter count.","headline":"A promising applied calibration paper whose real-network result is undermined by an unaddressed identifiability problem and an untested transfer-learning claim.","tokens_in":14255,"tokens_out":2033,"would_cite":false,"duration_ms":18773,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neuroevolution plus expert rules fits a real water grid's 20,000 parameters.","keywords":["Artificial Neural Networks","Automatic Calibration","Digital Twins","Full-Network calibration","Water Distribution Network Model","Neuro-Evolution of Augmenting Topologies","Expert Systems","Transfer Learning"],"falsifier":"Run ES-NEAT on the Fossolo benchmark, where the true parameter values are known, and compare the recovered roughness, demand, and leak coefficients against the synthetic truth: if the method reaches low flow-and-pressure RMSE while returning parameter values far from the truth, the central claim of global calibration fails even though the fit looks good.","tokens_in":13255,"feed_emoji":"💧","tokens_out":12517,"duration_ms":96629,"temperature":0.7,"pith_summary":"The paper introduces ES-NEAT, an automatic calibration method for drinking water distribution network models, and claims it solves a problem long treated as ill-posed: estimating tens of thousands of hydraulic parameters from very few field measurements. On a real Belgian network with 5,197 nodes and 4,158 pipes, the authors calibrate roughly 20,000 parameters (base demands, leaks, pipe roughnesses, minor and valve losses) using only 42 measured nodes and 8 measured pipes. They report a calibration RMSE of 0.33, a validation RMSE of 0.56 on five held-out locations, and a total cost of 10,000 EPANET simulations, about 12 hours on a laptop-class machine. The method's distinct promise is that the evolved neural network stores the calibration result as a structured set of inter-parameter relationships, so a later recalibration can start from the previous network instead of from scratch. The paper aims to establish both a practical recipe for full-network calibration under data scarcity and a mechanism for transferring calibration knowledge across the repeated recalibrations that real water utilities face.","feed_headline":"Neuroevolution plus expert rules fits water grid's 20,000 parameters","feed_subtitle":"Sparse sensors and 10,000 simulations reach RMSE 0.33, with the trained network saved for faster recalibration.","key_machinery":"The load-bearing machinery is ES-NEAT, a combination of an expert system with NEAT (Neuro-Evolution of Augmenting Topologies), a genetic algorithm that evolves both the weights and the topology of neural networks. In this application the neural network takes known network attributes such as pipe lengths, connectivity, diameters, and elevations as inputs and outputs calibrated parameters such as base demands, leak coefficients, pipe roughnesses, minor losses, and valve losses; NEAT searches over network topologies and weights, while the expert system supplies the rules that define each parameter's plausible range and prior distribution from stakeholder knowledge. Because flow and pressure respond to overlapping sets of parameters, the framework separates the calibration into a flow-sensitive phase followed by a pressure-sensitive phase, iterating until convergence, to stop one parameter silently compensating for another. The final ANN topology with its connection weights is the repository of calibration information that the paper proposes to transfer to subsequent calibration runs.","core_discovery":"The central claim is that ES-NEAT achieves accurate, global calibration of a real DWDN model with about 20,000 parameters in a setting where the parameters outnumber the measurements by orders of magnitude. The framework first runs a benchmark comparison of established optimization methods on the Fossolo network and selects NEAT as the best trade-off among accuracy, convergence speed, and the ability to store knowledge; ES-NEAT then pairs NEAT with an expert system whose rules constrain each parameter's range and distribution using stakeholder and domain knowledge. Calibration proceeds sequentially: flow-sensitive parameters such as base demands and leaks are adjusted first, then pressure-sensitive parameters such as roughness and minor losses, which the paper argues reduces the compensatory drift that makes underdetermined calibration meaningless. The reported outcome on the real network is an RMSE of 0.33 for calibration and 0.56 for validation, achieved with 10,000 simulations in about 12 hours. The paper further claims that the trained neural network encodes the calibrated parameter interactions and can be reused as the initial generation of a future NEAT run, so subsequent calibration efforts require less search.","pith_inferences":["Inference: if the stored ANN genuinely carries interaction structure, it could be read as a data-driven identifiability map: parameters with near-zero connection weights are candidates for being unobservable from the current sensor layout, which utilities could use to plan where to add meters.","Inference: the transfer-learning component is the natural next experiment: take the trained network from the Belgium case, alter one sector of the network, recalibrate from the stored ANN, and compare generations-to-convergence against a cold start; the paper implies but does not run this test.","Inference: because the expert system's rules set parameter ranges, ES-NEAT's accuracy depends on those ranges being right; an ablation that replaces stakeholder rules with wide uniform priors would separate the contribution of expert knowledge from that of NEAT itself.","Inference: the validation protocol, five held-out locations, tests interpolation between nearby sensors; a stronger generalization test would hold out an entire pressure zone or a different season, since spatially correlated errors could survive the current check."],"forward_implications":["Calibration of large real DWDNs can be completed in about 12 hours with only a few dozen measurement points, a cost that makes automated recalibration feasible on a routine basis.","The stored ANN gives a subsequent calibration run a structured starting point, so repeated calibrations over a network's lifetime should require less computation than the first calibration.","The benchmark comparison identifies NEAT as offering the best accuracy-convergence trade-off among the tested methods, which justifies using neuroevolution rather than a generic genetic algorithm for the real-network calibration.","Separating flow- and pressure-sensitive parameters reduces inter-parameter compensation, so the calibrated parameter set is expected to be more physically consistent than one produced by a single simultaneous search."],"supporting_citations":[{"why":"Defines NEAT, the neuroevolution algorithm whose topology search is the core of ES-NEAT.","marker":"Stanley & Miikkulainen, 2002"},{"why":"Provides the expert-systems methodology used to encode stakeholder rules and parameter constraints.","marker":"Kaisler, 1986"},{"why":"Supplies EPANET, the hydraulic simulator that evaluates every candidate parameter set.","marker":"Rossman, 2020"},{"why":"Supplies the Python-EPANET interface used to run the model within the optimization loop.","marker":"Klise et al., 2017"},{"why":"Provides the Fossolo benchmark network used to compare candidate calibration methods.","marker":"Wang et al., 2015"},{"why":"The ANN-plus-PSO calibration baseline that ES-NEAT is compared against and positioned relative to.","marker":"Meirelles et al., 2017"},{"why":"The genetic-algorithm calibration approach used as the second comparative baseline.","marker":"Nicolini et al., 2010"},{"why":"Defines transfer learning, the mechanism the paper invokes for reusing the trained ANN in later calibrations.","marker":"Tan et al., 2018"},{"why":"Prior work on which the ES-NEAT methodology is developed.","marker":"Gomez et al., 2021"}],"fun_headline_variants":["Expert rules plus neuroevolution fit water grid's 20k parameters","Sparse sensors calibrate 20k water parameters in 12 hours","Reusable neural net cuts water recalibration effort","Neuroevolution and expert systems calibrate large water models","Water network calibration from sparse data with reusable AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 42 pressure nodes and 8 flow pipes scattered through a 5,197-node, 4,158-pipe network carry enough information to pin down about 20,000 demand, leak, roughness, and loss parameters; if the measurements cannot identify those parameters, the low RMSE may come from compensating errors rather than from a correct calibrated state.","fun_headline_variants_meta":{"raw":{"variants":["Expert rules plus neuroevolution fit water grid's 20k parameters","Sparse sensors calibrate 20k water parameters in 12 hours","Reusable neural net cuts water recalibration effort","Neuroevolution and expert systems calibrate large water models","Water network calibration from sparse data with reusable AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2943,"prompt_tokens":1062,"completion_tokens":1881,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":1800}},"tokens_in":678,"tokens_out":1881,"duration_ms":12625,"temperature":1.0,"reasoning_tokens":1800,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:13:11.663964+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ES-NEAT on the Fossolo benchmark, where the true parameter values are known, and compare the recovered roughness, demand, and leak coefficients against the synthetic truth: if the method reaches low flow-and-pressure RMSE while returning parameter values far from the truth, the central claim of global calibration fails even though the fit looks good.","supporting_citations":[],"review_version":1}