{"id":"fda6fa4e-1def-41b5-9416-d533526843ad","arxiv_id":"2501.14503","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new set of 56 UAV path-planning benchmark instances is introduced and shown via landscape analysis to differ from standard suites, with CEC-winning evolutionary methods performing best.","lead":"This paper creates 56 new UAV path-planning benchmark instances and tests 12 global optimization methods on them. The generated instances differ from standard benchmark suites, and recent evolutionary algorithms from CEC competitions rank best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The per-setting best-found baseline used for all relative errors (Sec. 4) is, on instances where no method finds a feasible path (e.g., terrain 32, Fig. 6d), an infeasible solution above the Jpen=1e4 cliff; rankings may then measure feasibility, not closeness to the unknown optimum.","rationale":"The reader's weakest assumption — that unknown global optima make the best-found reference potentially unreliable — is exactly the load-bearing premise. The paper's own Fig. 6d provides direct evidence that this premise fails on at least one instance, and the Jpen cliff in Eq. 2 gives a concrete mechanism by which relative-error rankings are distorted by feasibility rather than by nearness to the optimum. This does not disprove the paper's conclusion, but it means the central claim currently rests on a reference baseline that is, for a subset of hard instances, an infeasible solution. The proposed check is a direct, low-cost way to determine whether the rankings change when a feasible reference is supplied. Because the reader already issued a CONDITIONAL verdict that identifies this assumption, my reading does not change the verdict; it adds a sharper, testable version of the same concern. I also note that the paper is otherwise careful: it provides code, data, and explicit limitations (Sec. 2.2.6), and it does not overclaim certainty, so the appropriate disposition remains conditional acceptance pending this verification.","tokens_in":26517,"tokens_out":4496,"duration_ms":43110,"concrete_test":"For each instance where no method achieved a feasible path (e.g., terrain 32 in Fig. 6d), construct a feasible path with a dedicated planner (visibility graph or A* on the same cost function) and verify its cost is below Jpen = 1e4. Use these feasible costs as the reference for those instances (keeping the per-setting best otherwise), then recompute mean relative error, number of wins, Friedman ranks, and the Wilcoxon/Holm tests. If EA4eig, APGSK, or ELSHADE cease to be consistently top-ranked, the central claim depends on an unreliable infeasible baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All quantitative conclusions (mean relative error, number of wins, Friedman ranks, Wilcoxon tests) are computed against the best solution found by any of the 12 methods in that setting (Sec. 4, first paragraph). This is a valid proxy only if the best-found solutions are close to the true optima. The objective function, however, contains a hard infeasibility penalty Jpen = 1e4 (Eq. 2, Sec. 2.2.2), creating a cliff: a path that barely clips a threat jumps from a cost of order hundreds to over 1e4. Relative errors are therefore dominated by whether a method finds any feasible path, not by how close to the optimum it is. The paper itself shows terrain 32 (Fig. 6d) where none of the 12 methods found a feasible solution although one exists; the per-setting best-found reference is then infeasible, and the relative errors for that instance rank degrees of infeasibility, not optimization quality. Since several hard instances may fall in this category, the reported superiority of EA4eig, APGSK, and ELSHADE could reflect their ability to cross the feasibility threshold under the fixed seed and budget, rather than a genuine ordering by objective-function proximity. This concern directly undercuts the abstract's claim that the best-ranking methods are 'almost universally' the top CEC evolutionary techniques. The paper's own stated limitation in Sec. 2.2.6 (height constraint checked only at waypoints) is separate but further reinforces that the objective landscape is discontinuous and that small path changes produce large cost jumps.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a benchmark suite of 56 UAV path-planning instances generated from 28 hand-picked terrains with two threat densities, and compares 12 global optimization methods (DIRECT-type, classical EC, and recent CEC competition winners) under four dimensionalities (DV=5,10,15,20) and three computational budgets (B=1e3,1e4,1e5). The authors use Exploratory Landscape Analysis (ELA) to argue that the instances are distinct from BBOB, CEC, and ABS benchmark functions, and they report mean relative errors, numbers of best-found solutions, Friedman ranks, and Wilcoxon tests with Holm correction. They find that EA4eig, APGSK, and ELSHADE are the best-performing methods overall, and they discuss the variable-dimension nature of the problem, concluding that solving in lower dimensions is generally more beneficial.","tokens_in":26797,"tokens_out":5676,"duration_ms":52419,"significance":"If the results hold, the proposed UAV instances could serve as a reusable real-world benchmark family, addressing the need for non-artificial test problems in global optimization. The paper provides a public Zenodo repository with code, data, and instance generation routines, which is a valuable asset for reproducibility. The finding that recent CEC-winning evolutionary algorithms outperform DIRECT-type deterministic methods on these discontinuous, constrained landscapes is of interest to both the evolutionary computation and path-planning communities. The variable-dimension investigation is a useful step toward a largely under-studied benchmark scenario. However, the validity of the quantitative conclusions depends on the choice of reference baseline and on the experimental design; these issues are addressed in the major comments.","major_comments":[{"comment":"The performance metric is defined as the relative error of a method's best-found solution to the best-found solution across all methods in that setting. Because the objective contains the hard infeasibility penalty Jpen=1e4, instances on which no method finds a feasible path (e.g., terrain 32 in Fig. 6d) have an infeasible reference solution with objective value above the cliff. On such instances, the relative errors measure degrees of infeasibility rather than closeness to the unknown optimum, and a method that finds a feasible path would have a negative relative error under the formula given in the text. Since all quantitative conclusions (mean relative error, number of wins, Friedman ranks, Wilcoxon tests) are computed against this baseline, the abstract's claim that the best-ranking methods are 'almost universally' the top CEC evolutionary techniques is not established for hard instances. The authors should report feasibility rates separately, compute relative errors against the best feasible solution found by any method across all settings, and verify whether the qualitative rankings survive this re-analysis.","section":"Section 4, first paragraph; Eq. (2)"},{"comment":"All methods are run once per instance and setting, starting from an identical random seed. For stochastic global optimizers, a single run provides no information about run-to-run variability, and the Wilcoxon signed-rank tests in Tables 2, 4, and 6 are based on a single paired sample per method, so the reported significance results may not be robust to the choice of seed. The authors should either run multiple independent repetitions (e.g., 10 or more) and report median/mean values with appropriate variance-aware statistical tests, or they should justify the single-run design by, for example, demonstrating low variance on a subset of instances. This is load-bearing because the central ranking claims depend on statistical significance assertions.","section":"Section 3.2"},{"comment":"The 56 instances are selected from a pool of 5000 by 'hand-picking twenty-eight terrains that displayed interesting and real-life looking characteristics' without quantitative selection criteria. This makes the instance selection subjective and not independently reproducible, and it may bias the ELA-based novelty analysis toward instances that appear unique by construction. The authors should either specify a quantitative selection procedure (e.g., based on terrain roughness statistics, obstacle coverage, or ELA feature values) or provide the full set of selection criteria so that readers can assess whether the claimed separation from BBOB/CEC/ABS suites is an artifact of hand-picking.","section":"Section 2.1; Section 2.3"}],"minor_comments":[{"comment":"There are several typos: 'AUV path planning' in Section 1 should be 'UAV path planning'; 'threads' appears in place of 'threats' in the discussion of Fig. 7 and in the text near Table 8; 'nad' in Section 2.2.2 should be 'and'; and the statistical test caption says 'Wilcoxon text' instead of 'Wilcoxon test'.","section":"Throughout"},{"comment":"The term 'relative error' is a misnomer when the reference is a best-found solution rather than a known optimum; consider using 'relative deviation from the best-found solution' or 'performance ratio' to avoid implying ground-truth error.","section":"Section 4, first paragraph"},{"comment":"The text says 'The only two methods for which no statistically significant difference from the best method (EA4eig) was found were APGSK (for DV=5) and LSHADE (for DV={5,15})' for B=1e3, but Table 2 shows LSHADE DV=15 with p*=0.0503, which is exactly the significance level; this borderline result should be reported with caution.","section":"Tables 2, 4, and 6"},{"comment":"In Table 7, the best-found solutions for each DV are taken from methods that may have used different budgets and may have been truncated by the three-hour limit for B=1e5; this should be stated more prominently so that the comparison across DV values is interpreted with the appropriate caveats.","section":"Section 4, variable dimension analysis"},{"comment":"The ELA computations use uniform sampling with 250*dim samples in dim=30, which is a relatively small sample for a 30-dimensional discontinuous objective; the authors should discuss whether the ELA features are stable with respect to sample size, especially given the Jpen=1e4 discontinuity.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmarking study with a useful public artifact (code, data, and instance generator) and follows published methodological guidelines. The major concerns are fixable: the authors can re-analyze the results using only feasible solutions or a common reference, add multiple runs, and make the instance selection reproducible. I do not see a fatal flaw in the concept, but the current evidence for the central ranking claim is weakened by the infeasible-best-found baseline and the single-run design. The manuscript fits the scope of the journal and should be reconsidered after a substantive revision addressing the load-bearing points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper gives the community a new, reusable UAV path planning benchmark with a generator, 56 hand-picked instances, an ELA-based comparison to established suites, and a careful comparison of 12 methods across dimensions and budgets. Code and data are public, and the experimental protocol follows recent methodological guidelines. That is a concrete contribution, and I'd rather see this published than another artificial function suite. The authors also deserve credit for flagging several limitations themselves, including the height constraint check and the behavior of the Jpen penalty.\n\nThe load-bearing concern is the one the stress-test hits. All relative errors, Friedman ranks, and win counts are computed against the best solution found by any method in that setting. The objective has a hard cliff at Jpen=1e4: any path that clips a threat jumps from a cost of hundreds to over 1e4. The paper itself shows a terrain (32) where none of the 12 methods found a feasible path, even though one exists. On such instances, the \"best-found\" reference is an infeasible solution, and the relative error metric ranks degrees of infeasibility rather than proximity to the true optimum. That does not make the paper's qualitative conclusion obviously wrong, but it does undercut the \"almost universally\" claim: EA4eig, APGSK, and ELSHADE may simply be better at crossing the feasibility threshold under a fixed seed and budget. The authors should either compute reference values from known-feasible solution classes, report the number of feasible solutions per method per instance, or at least explicitly separate feasibility success from objective-function quality in the rankings.\n\nTwo smaller issues: the methods are run once per setting, so there are no variance estimates for stochastic algorithms; and the 56 instances are hand-picked with no quantitative criteria for representativeness. Neither is fatal, but both limit how far the rankings can be generalized. The variable-dimension analysis is a nice addition, though it uses the same best-found baseline problem.\n\nWho is this for? Researchers benchmarking derivative-free optimizers, especially those interested in real-world problem instances. It deserves serious peer review — with the baseline issue fixed or honestly discussed, it would be a useful benchmark paper. I'd send it out.","headline":"The UAV benchmark suite is a real addition to the field, and the comparison is mostly sound, but the best-found baseline with a hard infeasibility penalty means the rankings partly measure feasibility, not objective-function quality.","tokens_in":27379,"tokens_out":2273,"would_cite":true,"duration_ms":20802,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that UAV path-planning instances, generated from random terrains with cylindrical threats, form a distinct and reusable real-world benchmark family, and that on these problems the top evolutionary algorithms from recent…","keywords":["unmanned aerial vehicle","path planning","global optimization","benchmarking","exploratory landscape analysis","variable dimension","evolutionary computation","DIRECT algorithm"],"falsifier":"For a given terrain, construct an instance whose optimal path is known (e.g., place all threats so the straight line from start to goal is feasible and optimal) and check whether the relative-error rankings of EA4eig, APGSK, and ELSHADE against the true optimum reproduce the rankings the paper reports against best-found solutions.","tokens_in":26262,"feed_emoji":"🛸","tokens_out":6276,"duration_ms":53503,"temperature":0.7,"pith_summary":"This paper argues that UAV path planning, formulated as a continuous black-box optimization problem with 56 hand-picked instances, is a useful addition to global optimization benchmark suites. The instances, generated from random terrains with cylindrical threats, have landscape features that differ clearly from established BBOB, CEC, and ABS problems. In computational comparisons across four dimensions and three budgets, the best performers are consistently top-ranked evolutionary methods from recent numerical-optimization competitions—EA4eig, APGSK, and ELSHADE—rather than deterministic DIRECT-type methods. The paper also shows that the problem's dimension can vary, and that most best-found solutions occur at the lowest dimension. If true, this gives the optimization community a reusable real-world benchmark family with a largely unexplored variable-dimension characteristic.","feed_headline":"Competition-winning evolutionary methods lead on a new UAV path benchmark","feed_subtitle":"The 56 generated instances look unlike existing benchmark suites, and evolutionary methods beat DIRECT-type solvers especially at large…","key_machinery":"The central object is a problem-instance generator producing 56 UAV path-planning instances: 28 randomly generated terrains, each with 15 or 30 cylindrical threats, combined with a weighted four-criterion cost function (path length, obstacle avoidance, altitude limits, and smoothness) whose decision variables are the waypoint coordinates. The argument is carried by comparing these instances to established benchmark suites through Exploratory Landscape Analysis (ELA) features, and by evaluating twelve solvers with relative error to the best-found solution, number of wins, and Friedman ranks across dimensions 15, 30, 45, 60 and budgets $10^3$, $10^4$, $10^5$ function evaluations.","core_discovery":"The central claim is that the generated UAV path-planning problems form a distinct, reusable benchmark family whose landscape characteristics are unlike those of the BBOB, CEC, and ABS suites, and that on these problems the best-performing methods are almost universally the top evolutionary algorithms from recent numerical-optimization competitions. At large computational budgets, EA4eig and APGSK consistently have the lowest Friedman ranks, with statistically significant advantages over most other methods, while ELSHADE, LSHADE, and SPSO also perform well in specific settings; deterministic DIRECT-type methods (BIRMIN, I-DTC-GL) and simple local search (NM) lag behind, especially as dimension and budget grow. The conclusion is stated on the paper's own terms: the UAV instances 'may form an interesting addition to the established benchmark suits.'","pith_inferences":["Because the global optima are unknown, the rankings are relative to the best-found solution; a direct extension would be to construct instances with known optimal paths (e.g., by placing threats so the straight line is optimal) to check whether the reported ordering holds under an absolute reference.","The variable-dimension finding suggests a practical recipe for UAV path planning: start with few waypoints to find a feasible corridor, then refine with more waypoints; this could be tested as a standalone 'variable-dimension' method and compared against fixed-dimension baselines.","The landscape-uniqueness result implies that these instances may expose failure modes hidden by artificial benchmarks; a concrete way to check this is to see whether methods tuned on established suites degrade less or more on UAV instances relative to their landscape-similar artificial functions.","Porting the MATLAB generator to other languages and to established profiling platforms would let the community measure per-run behavior (e.g., anytime curves) rather than only final errors."],"forward_implications":["The 56 UAV instances are a reusable benchmark family: code and data are publicly released, so other researchers can run the same problems without re-implementing the generator.","On these instances, method rankings depend on both dimension and computational budget, so benchmarks that test only a single budget or dimension will miss the patterns.","The variable-dimension view of the problem favors starting at low dimension ($D_V=5$) and refining: in most cases best-found solutions come from the lowest-dimension setting, and higher dimensions rarely compensate for added difficulty.","For large budgets, the performance gap between many methods narrows, but APGSK and EA4eig remain consistently ahead.","High-density threat instances are harder and show larger improvements when the budget grows, because methods need more evaluations to escape infeasible or locally optimal corridors."],"supporting_citations":[{"why":"Introduces the spherical vector-based particle swarm optimization (SPSO) safety-enhanced UAV path planning model that the paper builds on for the optimization formulation.","marker":"[9]"},{"why":"Supplies the four-criteria objective function (path length, obstacle avoidance, altitude, smoothness) adapted in Section 2.2.","marker":"[59]"},{"why":"Terrain generation tool used to create the 28 terrains from which the 56 instances are derived.","marker":"[58]"},{"why":"Provides the established benchmark suites used for the Exploratory Landscape Analysis comparison.","marker":"[60]"},{"why":"Provides the R package used to compute the ELA features that support the uniqueness claim.","marker":"[61]"},{"why":"Recent large benchmarking study from which the twelve state-of-the-art global optimization methods are selected.","marker":"[64]"},{"why":"Methodological guidelines for budgets, statistical tests, and reporting that structure the experimental comparison.","marker":"[65]"},{"why":"Public repository for the instance generation code, experiment implementations, and resulting data, supporting the reusable-benchmark claim.","marker":"[56]"}],"fun_headline_variants":["UAV path planning offers fresh benchmark for global optimizers","Evolutionary solvers outpace DIRECT on UAV path benchmarks","New UAV benchmark: evolutionary methods dominate DIRECT","Distinct UAV benchmark favors top evolutionary optimizers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The global optima of the 56 instances are unknown, so all relative-error metrics and rankings are computed against the best solution found by any method on each instance; if those best-found solutions are far from the true optima, or if all methods fail similarly on hard instances, the reported rankings may not reflect true performance.","fun_headline_variants_meta":{"raw":{"variants":["UAV path planning offers fresh benchmark for global optimizers","Evolutionary solvers outpace DIRECT on UAV path benchmarks","New UAV benchmark: evolutionary methods dominate DIRECT","Distinct UAV benchmark favors top evolutionary optimizers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000449,"raw_usage":{"total_tokens":2236,"prompt_tokens":892,"completion_tokens":1344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1282}},"tokens_in":508,"tokens_out":1344,"duration_ms":8850,"temperature":1.0,"reasoning_tokens":1282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:05:16.114439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a given terrain, construct an instance whose optimal path is known (e.g., place all threats so the straight line from start to goal is feasible and optimal) and check whether the relative-error rankings of EA4eig, APGSK, and ELSHADE against the true optimum reproduce the rankings the paper reports against best-found solutions.","supporting_citations":[{"cited_title":"Cheng, J","cited_arxiv_id":null,"evidence_quote":"Supplies the four-criteria objective function (path length, obstacle avoidance, altitude, smoothness) adapted in Section 2.2."},{"cited_title":"McClure, Automatic terrain generation, matlab central file exchange, 2023","cited_arxiv_id":null,"evidence_quote":"Terrain generation tool used to create the 28 terrains from which the 56 instances are derived."},{"cited_title":"com/blockchain-group/DIRECTGOLib, pre-release v2.0","cited_arxiv_id":null,"evidence_quote":"Provides the established benchmark suites used for the Exploratory Landscape Analysis comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the R package used to compute the ELA features that support the uniqueness claim."},{"cited_title":"Stripinis, J","cited_arxiv_id":null,"evidence_quote":"Recent large benchmarking study from which the twelve state-of-the-art global optimization methods are selected."},{"cited_title":"LaTorre, D","cited_arxiv_id":null,"evidence_quote":"Methodological guidelines for budgets, statistical tests, and reporting that structure the experimental comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Public repository for the instance generation code, experiment implementations, and resulting data, supporting the reusable-benchmark claim."}],"review_version":1}