{"id":"490a18fa-4592-4ccd-8773-aad5a5c58aeb","arxiv_id":"2507.16327","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"WPgen uses NSGA-II to generate minimal waypoint perturbations that make maritime vessels navigate in unstable paths, to help test self-adaptation software.","lead":"This paper presents WPgen, a method that slightly modifies the waypoints a maritime autonomous vessel is supposed to follow, to make the vessel navigate badly. The goal is to find waypoint changes that help designers test whether the vessel's self-adaptation software reacts correctly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The link between the Eq. (3) fitness proxy and the autocorrelation-based instability classifier is never validated, so the central claim that WPgen generates waypoints that trigger adaptations rests on an unverified surrogate.","rationale":"I read the paper as making a specific, testable claim: WPgen's multiobjective search over waypoint perturbations produces waypoint sets that lead to unstable navigation, and such unstable navigation is the intended trigger for self-adaptation. The load-bearing link in that argument is the fitness function of Eq. (3). The paper defines two notions of instability: one in the search objective (path-length ratio) and one in the evaluation and reporting (autocorrelation of angle measurements with a 0.1 threshold, Sect. 4.3.3). The two are never connected. This is not merely a stylistic mismatch; it is a validation gap that determines whether the empirical results answer RQ1 as posed. The RQ1 comparison is conducted in HV space over the two objectives including Eq. (3), so a large-effect-size HV improvement (Table 1) shows only that WPgen finds long/deviating paths close to the original waypoints. The paper's own RQ3 results show that for the underwater AVs, the majority of Pareto-front solutions are classified stable (68-80%), so the connection between the optimized objective and the classifier is at best weak and at worst absent. The reader's weakest_assumption identifies exactly this proxy-validation gap, and I agree with the CONDITIONAL verdict. The concrete test is straightforward with the provided replication package: collect all evaluated individuals and their simulator outputs, compute Eq. (3) and the autocorrelation classification, and measure the correlation. If the correlation is high, the concern is dissolved; if low, the paper's central claim needs rework. I do not see a more load-bearing issue: the statistical methodology is sound, the replication package exists, and the empirical honesty (reporting RS beating WPgen_seed on Remus100) supports good faith. The novelty of the problem formulation is real. The missing MAPE-K trigger demonstration is part of the same gap, because the adaptation trigger presumably depends on the classifier or on some runtime monitor; but the proxy-classifier link is the more direct and more testable weakness.","tokens_in":15380,"tokens_out":2133,"duration_ms":20478,"concrete_test":"Compute, for each of the three AVs, the Spearman rank correlation between fitunstable (Eq. 3) and the binary unstable classification (Sect. 4.3.3) over the union of all evaluated individuals across the 30 runs of WPgen_rnd. If the rank correlation is below, say, 0.5 for Remus100 or Nspauv, the fitness proxy is not a faithful surrogate and the HV-based comparison in RQ1 does not support the claim that WPgen outperforms RS at generating instability-triggering waypoints. Additionally, rerun the RQ1 comparison using the fraction of classifier-unstable sub-paths per solution as the objective; report whether the WPgen-vs-RS advantage survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that WPgen generates minor waypoint modifications that cause unstable navigation and thereby provide test scenarios for triggering self-adaptations. The search is guided by fitunstable in Eq. (3), defined as the sum over consecutive waypoint pairs of the ratio of simulated path length |sp_i| to the straight-line distance between original waypoints. Instability, however, is measured and reported in Sect. 4.3.3 / Table 2 with a separate classifier: autocorrelation of vessel angles with a threshold of 0.1, with a sub-path labeled unstable if at least one angle is unstable. Nowhere does the paper show that maximizing Eq. (3) correlates with this classifier. In fact, the two measures need not agree: a path can be long and deviating (high Eq. (3)) while the vessel's roll/pitch angles remain smooth and autocorrelated, or short yet oscillatory. Table 2 reports that high-fitness Pareto-front solutions are classified unstable only 18-25% of the time for the underwater vessels and about 82-83% for Mariner; but this is a post-hoc fraction across all Pareto solutions, not a validation that optimized Eq. (3) values predict the unstable class. Because RQ1's HV comparisons are computed on the Eq. (3) objective space (plus distance), a significant HV improvement does not entail more unstable paths under the classifier. Consequently, the strongest claim about triggering adaptations is undercut: if the proxy is weakly correlated with the classifier, the generated waypoint sets may simply be 'long detours' rather than instability-inducing scenarios. The paper also never demonstrates that any classified-unstable path actually triggers a MAPE-K adaptation, a step the authors defer to future work.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents WPgen, a multi-objective search-based approach for generating minor modifications to predefined waypoints of maritime autonomous vessels, with the aim of producing unstable navigation behavior that should trigger self-adaptation. WPgen uses NSGA-II with three seeding strategies and two objectives: minimizing the deviation from the original waypoints (Eq. (2)) and maximizing a path-length-based instability proxy (Eq. (3)). The approach is evaluated on three vessel models in the Marine Systems Simulator against random search, using hypervolume as the quality indicator and an autocorrelation-based classifier for path stability (RQ3). The results show that WPgen variants significantly outperform random search on the Mariner tanker, are mixed on the underwater vessels (with random search beating the fully seeded variant on Remus100), and are not significantly different on Nspauv.","tokens_in":15583,"tokens_out":5940,"duration_ms":59919,"significance":"If the connection between the Eq. (3) proxy and actual vessel instability were established, WPgen would be a useful contribution to scenario-based testing of self-adaptive maritime navigation software, addressing a real need within the RoboSAPIENS project. The paper's strengths include the use of a recognized maritime simulator (MSS), three realistic vessel models, standard statistical methodology (Mann-Whitney U, Vargha-Delaney A12, 30 runs), a replication package, and clear reporting of results. However, the central claim that WPgen generates waypoints that trigger self-adaptations is not yet supported, because the search objective is an unvalidated surrogate for the instability criterion used in evaluation.","major_comments":[{"comment":"Section 3.4, Eq. (3) defines fit_unstable as the sum of ratios of simulated sub-path lengths to straight-line distances between original waypoints, but Section 4.3.3 defines instability operationally via autocorrelation of vessel angles with a threshold of 0.1. The manuscript never demonstrates that maximizing Eq. (3) is correlated with this classifier; a long smooth deviation and a short oscillatory path can score oppositely on the two measures. Since RQ1 and RQ2 and the HV comparisons are computed in the Eq. (2)-Eq. (3) objective space, significant hypervolume improvements do not entail that WPgen produces more waypoints that would be classified as unstable. The authors should validate the proxy, for example by reporting the correlation or classification agreement between Eq. (3) and the autocorrelation labels, or by using the classifier as the fitness function.","section":"Section 3.4, Eq. (3); Section 4.3.3"},{"comment":"Section 4.3.3 describes the stability classifier only in general terms: it is not stated which axes are measured, at what sampling rate autocorrelation is computed, how the peak threshold of 0.1 was determined, or how the 'more than one threshold peak' rule was validated. Reference [26] is a general discussion of correlation coefficients and does not justify this specific threshold for vessel-angle autocorrelation. Without these details, the RQ3 classification results cannot be independently reproduced or assessed, which is important because that classifier is the only operational definition of 'unstable' in the paper.","section":"Section 4.3.3"},{"comment":"Section 5.3, Table 2 reports percentages of stable, unstable, and missing sub-paths across all Pareto-front solutions (e.g., 18%-25% unstable for the underwater vessels and around 82%-83% for Mariner). These are descriptive post-hoc rates; they do not validate that high values of Eq. (3) predict the unstable class. A proper validation would compare the distribution of Eq. (3) values for classifier-labeled stable versus unstable sub-paths, or report precision and recall of the proxy at relevant thresholds.","section":"Section 5.3, Table 2"},{"comment":"Section 5.1, Table 1 and Figure 3: the RQ1 conclusion that the problem is complex enough to require a guided algorithm is based on HV computed in the surrogate objective space. On Nspauv there are no significant differences, and on Remus100 random search significantly outperforms WPgenseed in HV; moreover, because the surrogate is unvalidated, even the Mariner advantage does not directly support the claim that WPgen triggers more self-adaptations.","section":"Section 5.1, Table 1 and Figure 3"}],"minor_comments":[{"comment":"Equation (3) sums over N-1 sub-paths, but the text notes that the vessel may fail to reach all waypoints, in which case there are fewer than N-1 sub-paths; the summation bound and the treatment of missing sub-paths in the fitness function should be clarified.","section":"Section 3.4, Eq. (3)"},{"comment":"The hyperparameter Delta is set to 400 for Mariner and 150 for the underwater vessels based on vessel characteristics, but the sensitivity of the results to this choice is not studied; a brief sensitivity discussion would strengthen the parameter-setting rationale.","section":"Section 4.2.3"},{"comment":"The statement that a full path can have 2^{N-1} + 2(N-2) different categorizations is unexplained and appears inconsistent with the three sub-path categories; the reasoning behind this count should be made explicit.","section":"Section 4.3.3"},{"comment":"Line 8 of Algorithm 1 is formatted ambiguously ('while p < 0.5 numOfMutParams'); it should be written as a clear condition, likely p < 0.5^{numOfMutParams} or p < 0.5 * numOfMutParams.","section":"Algorithm 1"},{"comment":"The fourth author's email address is identical to the second author's (aarrieta@mondragon.edu); this appears to be a typographical error.","section":"Author information"},{"comment":"The sentence 'For Remus100, WPgen proved to be the best' is ambiguous because Table 1 shows that the choice of variant matters (WPgenseed loses to RS while WPgencomb and WPgenrnd beat RS); the text should identify which variant is meant.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a software engineering journal, and the replication package is a notable strength. The main risk is the unvalidated fitness proxy: the search objective in Eq. (3) and the evaluation classifier in Section 4.3.3 are different operationalizations of 'instability,' and the paper does not show that they agree. This is a fixable issue through additional validation experiments, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper poses a genuinely new test-generation problem and the evaluation is honest, but the central claim is not yet supported. WPgen is, as far as the related work shows, the first search-based approach for producing minor waypoint perturbations that make maritime autonomous vessels navigate poorly, with the eventual goal of testing self-adaptation logic. That problem formulation is worth referee time.\n\nWhat it does well: clean encoding, two sensible objectives (closeness to the original waypoints and a path-length-based instability proxy), three seeding strategies, and a fair comparison against random search using 30 runs, Mann-Whitney U, and A12 effect sizes. The results are reported without spin—on Remus100 the fully seeded variant is significantly worse than random search, and on Nspauv nothing beats random search. They also ship a replication package. That counts.\n\nThe soft spot is exactly the one the stress-test note flags. The search optimizes Eq. (3), the ratio of simulated sub-path length to straight-line distance between original waypoints. The reported outcome, however, is a different classifier: autocorrelation of vessel angles with a threshold of 0.1, chosen after an exploratory study that is never documented. The paper never shows that optimizing Eq. (3) yields sub-paths the autocorrelation classifier labels unstable. The post-hoc percentages in Table 2 do not close that gap; for Remus100 and Nspauv only around 18-25% of Pareto-front sub-paths are unstable, so the generated sets may often be long detours rather than instability-inducing scenarios. On top of that, no MAPE-K loop is ever simulated, so \"triggering self-adaptations\" is inferred, not demonstrated. The authors say the actual implementation is future work; that is honest, but it means the title and abstract overreach.\n\nNone of this is fatal. A direct correlation check between Eq. (3) and the autocorrelation classifier, a documented threshold selection, and one simulation showing a classified-unstable path actually triggering an adaptation would largely fix it. The related work is appropriate, and the novelty claim is scoped correctly. One smaller overstatement: the conclusion that the problem \"requires\" NSGA-II is not supported by Nspauv, where no method beat random search.\n\nFor me the verdict is conditional acceptance: send this to a serious referee, not a desk reject. The method is coherent, the experiments are reproducible, and the missing validation is a well-defined revision rather than a dead end.","headline":"A genuinely new waypoint-perturbation test generator, evaluated honestly but with an unvalidated fitness proxy and no direct trigger demonstration—worth a serious referee, conditional on closing that gap.","tokens_in":16252,"tokens_out":3385,"would_cite":false,"duration_ms":38828,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WPgen generates minor waypoint modifications that steer maritime autonomous vessels into unstable navigation, offering test scenarios for triggering and validating self-adaptation systems.","keywords":["autonomous vessels","self-adaptation","waypoint generation","search-based software testing","NSGA-II","multi-objective optimization","simulation-based testing","maritime software"],"falsifier":"Run WPgen on any of the three vessels to produce a Pareto front of modified waypoints, simulate each solution, and compute the fraction of sub-paths that the autocorrelation-based instability classifier marks as unstable. If that fraction is no higher for WPgen solutions than for randomly generated waypoints of the same distance from the original route, then the Eq. (3) objective is not actually steering the search toward unstable navigation and the central claim fails.","tokens_in":15103,"feed_emoji":"🚢","tokens_out":6300,"duration_ms":58996,"temperature":0.7,"pith_summary":"This paper argues that minor modifications to the waypoints guiding a maritime autonomous vessel can be generated automatically, and that those small edits are enough to provoke the kind of unstable navigation that should trigger the vessel's self-adaptation logic. The proposed approach, WPgen, casts waypoint generation as a two-objective search problem: keep the new waypoints as close as possible to the original route while maximizing a measure of path instability. Experiments on three simulated vessels show the approach outperforms random search with large effect size on the overwater tanker, while on the two underwater vessels the outcome depends on how the initial population is seeded. The practical aim is to give developers test scenarios for validating that their self-adaptation implementations trigger correctly before deployment.","feed_headline":"Slight waypoint edits can make autonomous ships veer unstable","feed_subtitle":"Automatically generated waypoint variations give developers test scenarios for validating vessel self-adaptation logic.","key_machinery":"The load-bearing mechanism is NSGA-II, a multi-objective evolutionary algorithm, configured to optimize two fitness functions: $\\text{fit}_{\\text{distWPs}}(\\text{WP})$ (Eq. 2), the Euclidean distance between original and modified waypoints, and $\\text{fit}_{\\text{unstable}}(\\text{WP})$ (Eq. 3), the sum over consecutive waypoint pairs of the ratio between the simulated path length and the original straight-line distance. The simulator supplies the vessel models and the simulated paths, a death-penalty filter enforces the minimum-distance constraint between consecutive waypoints, and three seeding strategies—fully seeded, combined, and fully random—define the three WPgen variants. The instability objective in Eq. (3) is the proxy that is supposed to align with the self-adaptation triggering condition, and its correctness is what the paper's utility ultimately rests on.","core_discovery":"On the paper's own terms, the central claim is that WPgen can produce waypoints that are minor variations of a vessel's predefined route and that nevertheless drive the navigation software into unstable behavior, giving testers a way to exercise self-adaptation mechanisms. The search is driven by two competing objectives: minimizing the Euclidean distance between the generated and original waypoints (Eq. 2) and maximizing the summed ratio of each simulated sub-path length to the straight-line distance between the corresponding original waypoints (Eq. 3). When evaluated on the Mariner overwater tanker, all three seeding variants of WPgen beat random search with large effect size; on the Remus100 underwater vessel only the combined and fully random seeding variants beat random search, while the fully seeded variant was significantly worse; and on the Nspauv underwater vessel no approach differed significantly from random search. The paper treats this vessel-dependence as the main empirical message: the choice of search strategy and seeding must be tuned to the vessel under test.","pith_inferences":["Because the search objective (Eq. 3) and the evaluation classifier (autocorrelation of vessel angles) are different measures, a natural next step is to validate their agreement; if they disagree, the generated tests may still fail to trigger adaptations even when the fitness function improves.","The same encoding and search operators could be applied to other waypoint-following robots, such as unmanned aerial vehicles, since the approach does not rely on maritime-specific dynamics beyond what the simulator provides.","The vessel-dependence of results suggests that a cheap pre-screening phase could estimate whether a given vessel's navigation software is search-testable, avoiding wasted computational effort on vessels like Nspauv where random search is sufficient."],"forward_implications":["For the overwater tanker Mariner, using any of the three WPgen seeding variants instead of random search is the recommended practice, since all three yield significantly better hypervolume with large effect size.","For underwater vessels, seeding the initial population with randomness (combined or fully random) is preferable to fully seeding it, because the fully seeded variant can be significantly worse than random search, as happened on Remus100.","The generated waypoints provide ready-made test scenarios for validating the Plan and Legitimate components of a MAPLE-K self-adaptation loop.","The sub-path classification (stable, unstable, missing) can be reused as a more direct fitness function for future search-based generation.","WPgen's effectiveness is vessel-dependent, so approach selection should be informed by the vessel model being tested."],"supporting_citations":[{"why":"Supplies NSGA-II, the multi-objective evolutionary algorithm that performs the search over waypoint modifications.","marker":"[22]"},{"why":"Supplies the evolutionary multi-objective optimization platform and default genetic operators used to realize WPgen.","marker":"[18]"},{"why":"Supplies the marine systems simulator and the three vessel models used as case studies.","marker":"[10]"},{"why":"Supplies the statistical testing protocol (30 runs, Mann-Whitney U, effect size) used to compare approaches.","marker":"[23]"},{"why":"Supplies the hypervolume indicator used to measure the quality of the Pareto fronts produced by each approach.","marker":"[24]"},{"why":"Supplies the autocorrelation-based instability classification used in RQ3 to categorize sub-paths as stable, unstable, or missing.","marker":"[26]"},{"why":"Supplies the death-penalty constraint handling that filters waypoints violating the minimum-distance constraint.","marker":"[14]"},{"why":"Supplies the MAPE-K self-adaptation reference loop that frames the need to find adaptation-triggering settings.","marker":"[1]"}],"fun_headline_variants":["Search-based waypoint tweaks stress-test autonomous ships","Waypoint mutations expose autonomous vessel adaptation flaws","Autonomous ship safety: minor route edits trigger instability","Vessel-specific waypoint search aids self-adaptation validation","WPgen: tweaking waypoints to test maritime self-adaption"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that the path-length ratio in Eq. (3) is a faithful proxy for the unstable navigation that should trigger a self-adaptation is the load-bearing premise, because the paper never shows that optimizing this ratio produces waypoints that the autocorrelation-based classifier would label unstable.","fun_headline_variants_meta":{"raw":{"variants":["Search-based waypoint tweaks stress-test autonomous ships","Waypoint mutations expose autonomous vessel adaptation flaws","Autonomous ship safety: minor route edits trigger instability","Vessel-specific waypoint search aids self-adaptation validation","WPgen: tweaking waypoints to test maritime self-adaption"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1483,"prompt_tokens":969,"completion_tokens":514,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":434}},"tokens_in":585,"tokens_out":514,"duration_ms":5602,"temperature":1.0,"reasoning_tokens":434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:12:09.964960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run WPgen on any of the three vessels to produce a Pareto front of modified waypoints, simulate each solution, and compute the fraction of sub-paths that the autocorrelation-based instability classifier marks as unstable. If that fraction is no higher for WPgen solutions than for randomly generated waypoints of the same distance from the original route, then the Eq. (3) objective is not actually steering the search toward unstable navigation and the central claim fails.","supporting_citations":[{"cited_title":"PlatEMO: A MATLAB platform for evolutionary multi- objective optimization.IEEE Computational Intelligence Magazine, 12(4):73–87, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the evolutionary multi-objective optimization platform and default genetic operators used to realize WPgen."},{"cited_title":"[Online; accessed 25-March-2025]","cited_arxiv_id":null,"evidence_quote":"Supplies the marine systems simulator and the three vessel models used as case studies."},{"cited_title":"Correlation coefficients: appropriate use and interpretation","cited_arxiv_id":null,"evidence_quote":"Supplies the autocorrelation-based instability classification used in RQ3 to categorize sub-paths as stable, unstable, or missing."},{"cited_title":"A review of constraint-handling techniques for evolution strategies.Appl","cited_arxiv_id":null,"evidence_quote":"Supplies the death-penalty constraint handling that filters waypoints violating the minimum-distance constraint."}],"review_version":1}