{"id":"31c4f103-5585-4a51-b05f-10e218adcfcc","arxiv_id":"2412.07655","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A closed-loop simulation and Bayesian optimization framework selects data augmentation hyperparameters for a YOLO landing-pad detector, improving simulated VTOL landing success from 50% to 70%.","lead":"The authors used Bayesian optimization to choose image augmentation settings (scale, brightness) for a YOLO landing-pad detector, retraining it based on landing success in a photorealistic VTOL simulator. In their simulations, the retrained model lifted landing success from 50% to 70% in clear conditions and held 70% at night and 50% in night rain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'at least 20% under different conditions' claim rests on 10-trial success rates and has no reported baselines for night or night-rain, so the improvement is not statistically established.","rationale":"The reader's weakest assumption was sim-to-real transfer. That is a real limitation, but the paper explicitly scopes its experimental validation to the CARLA/GUAM simulator and lists real-flight testing as future work (Section V); judged on its own terms, the central claim is about simulated landing success. The more pressing threat is internal to the reported numbers: with only 10 trials, the headline 50% to 70% jump is not statistically significant, and the BO procedure selects the best of 30 noisy evaluations. Missing baselines for clear night and night-with-rain make the 'consistently at least 20% under different conditions' claim even harder to defend. I therefore agree with the conditional verdict, but for a more internal reason than the sim-to-real proxy. A modest amount of additional simulation (100 trials per condition, paired seeds, and explicit baselines for the two non-day conditions) would either substantiate or refute the claim, so conditional rather than reject remains appropriate.","tokens_in":14055,"tokens_out":6861,"duration_ms":60697,"concrete_test":"For each of the three conditions (clear day, clear night, night rain), fix 100 random initial positions and run the baseline YOLO model and the final shared-parameter model on the identical seed-controlled trials; report per-condition success counts, Wilson 95% confidence intervals, and the baseline night and night-rain rates, which are currently absent. If the 95% CI for the baseline-vs-optimized difference includes 0 in any condition, or if the night-rain optimized rate is not above its baseline, the 'at least 20% across conditions' claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central quantitative claim ('at least 20% improvement under different lighting and weather conditions') rests on a single observed jump from 50% to 70% in Section IV.B. Every reported success rate is 5 or 7 successes out of 10 attempts (Section III.C), with no repeated trials, error bars, or confidence intervals. With 10 trials per condition, a 50%-vs-70% difference is well within sampling noise (two-tailed Fisher exact p ≈ 0.65 for independent 10-trial runs), so the improvement is not statistically established. The problem is compounded by the optimization loop: Algorithm 1 uses these noisy 10-trial evaluations as the GP objective, and the 'best' model is selected after 30 iterations, so the quoted 70% is a maximum over noisy evaluations (winner's curse), not an unbiased estimate. Additionally, Section IV.B gives a baseline only for clear day; no baseline success rates are reported for clear night or night-with-rain, yet the abstract and Section III.E claim consistent 20% improvements under those conditions. The final shared-parameter model achieves 50% at night with rain, which cannot be compared to any stated baseline. I also note that the GP contour plots show predicted success rates above 1.0 (Figure 9a, 9c), so the surrogate model used for acquisition is not calibrated as a probability. Thus the broad 'under different lighting and weather conditions' claim is not supported by the data as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a closed-loop framework for improving a YOLOv8 perception model used for autonomous VTOL landing. The framework combines photorealistic CARLA simulation with JAX-GUAM vehicle dynamics, a PID landing controller, and Bayesian optimization over two data-augmentation hyperparameters (scale and brightness) to retrain the detector against simulated landing success. The authors report that the initial model achieves a 50% landing success rate in clear day conditions, and that after 30 iterations of Bayesian optimization the success rate rises to 70%. They further report a shared parameter set (scale=0.77, brightness=0.66) that yields 70% success in clear day and clear night, and 50% success at night with rain. Real-flight testing is explicitly deferred to future work.","tokens_in":14262,"tokens_out":5964,"duration_ms":49052,"significance":"If the reported improvements are robust, the framework is a useful integration of photorealistic simulation, high-fidelity vehicle dynamics, perception, and Bayesian optimization for aerial landing tasks. The authors make their code available, and the decision to disable stochastic YOLO augmentations for reproducible training is a genuine strength. However, the current evidence is insufficient to support the headline quantitative claims: the central comparison rests on single ten-trial evaluations without variance or confidence intervals, baseline performance is only reported for one condition, and the paper contains an internal inconsistency about per-condition success rates. These issues are fixable within the manuscript's scope, so the framework itself appears promising rather than fundamentally flawed.","major_comments":[{"comment":"The claim that the model 'consistently improved the perception-based landing success rate by at least 20% under different lighting and weather conditions' is not supported by the reported data. The 50% to 70% improvement is based on one set of ten trials for the baseline and one set for the optimized model, with no repeated seeds, no variance, and no confidence intervals. For 5/10 versus 7/10 successes, a two-tailed Fisher exact test gives a p-value of roughly 0.6, so the observed difference is well within sampling noise. Moreover, Section IV.B reports a baseline only for clear day; no baseline success rates are given for clear night or night-with-rain, so the 'at least 20% under different conditions' claim cannot be verified for those conditions.","section":"Abstract and Section IV.B"},{"comment":"There is an internal contradiction in the reported results. Section III.E states that 'We obtained models that consistently perform with more than 70% landing success rates for each condition,' but Section IV.B reports that the final shared-parameter model achieves a 50% landing success rate in night-with-rain conditions. The authors should reconcile these statements and clearly distinguish the per-condition optimized models from the single shared-parameter model.","section":"Section III.E and Section IV.B"},{"comment":"The Gaussian process surrogate is used to model the landing success rate, but the contour plots in Figure 9a and 9c show predicted success rates greater than 1.0. Since a success rate is a probability, the GP is not a calibrated model of the objective, and the UCB acquisition function may query parameter regions with impossible predictions. The authors should either use a link function appropriate for binary or proportional outcomes (e.g., a logistic GP) or otherwise constrain the surrogate to [0,1], and they should discuss how the reported optimal parameters depend on this modeling choice.","section":"Section III.D and Figure 9"},{"comment":"The reported 70% success rate is the maximum over the 30 Bayesian-optimization iterations, not an unbiased estimate of the performance of the final selected model. This winner's-curse effect means the quoted improvement is likely inflated relative to the expected performance of the chosen hyperparameters on a fresh evaluation. The authors should re-evaluate the final selected hyperparameters (scale=0.77, brightness=0.66) on multiple independent sets of landing trials and report the mean and variance (or a confidence interval) for both the baseline and the optimized model.","section":"Algorithm 1 and Section IV.B"}],"minor_comments":[{"comment":"The 80/20 train-validation split and the five data subsets are described as random, but no random seed or reproducibility mechanism is specified for these splits; please provide the seeds or the code configuration so that the claimed reproducibility can be assessed.","section":"Section III.A"},{"comment":"The caption says '20% Improvement in Landing Success Rate,' but the improvement is 20 percentage points (50% to 70%), not 20% relative; please reword for accuracy.","section":"Figure 8 caption"},{"comment":"The sentence comparing mAP50-95 of 43% to YOLOv8's 44.9% on COCO should clarify that this is the initial model before applying the proposed framework, and should state whether the final model's detection metrics are also reported.","section":"Section IV.B"},{"comment":"The PID gains Kp, Ki, and Kd are not specified; please provide the numerical values used in the simulations so that the experiments could be reproduced.","section":"Section III.C"},{"comment":"The axis label 'Brightness Value (hsv_v)' is not defined in the text; please define brightness as the V channel of the HSV color space and state the augmentation range used.","section":"Figure 9"},{"comment":"The schematic includes a step labeled 'Validation: Address sim2real GAP,' but real-world deployment is only mentioned as future work; the caption should be updated to reflect that this step is not performed in the present study.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The framework is a reasonable contribution, and the authors' effort to make training reproducible and to release code is commendable. For a journal submission, however, the statistical quality of the central experiment needs to be substantially improved: repeated trials, error bars or confidence intervals, and baselines for all reported conditions are necessary. The internal inconsistency between Section III.E and Section IV.B should also be corrected before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rough take: this is a solid engineering integration paper, not a scientific breakthrough. The authors couple Bayesian optimization over two augmentation parameters (scale, brightness) with closed-loop landing success in a photorealistic CARLA/GUAM simulator to retrain a YOLOv8 detector for VTOL vertical landing. That closed-loop use of landing performance to drive augmentation is genuinely new in this subfield, as far as the citations go. The framework is modular, the training is made reproducible by disabling YOLO's random augmentation, and the code is linked. I believe the integration works as described.\n\nThe problem is the evidence behind the abstract's central claim. The 'at least 20% improvement under different lighting and weather conditions' rests entirely on one comparison: 50% to 70% success in clear day, each from ten trials. With ten trials, a two-category difference of 5 versus 7 successes is well within sampling noise (Fisher exact p≈0.65). The 70% figure is also the maximum over 30 noisy evaluations, so winner's curse inflates it. No repeated seeds, no confidence intervals. For clear night and night-with-rain, the paper reports final rates of 70% and 50% but gives no baseline at all; the abstract's 'at least 20%' under those conditions has no visible support. The GP contour plots in Figure 9 predict success rates above 1.0, meaning the surrogate model used for acquisition is not calibrated as a probability.\n\nThe body text is more honest: it presents the 50-to-70 as the experimental result and says real-flight testing is future work. So the discrepancy is mostly in the abstract's overreach, not in the body's claims. The optimization loop itself is standard GP-UCB; no mathematical novelty there, but that's not a flaw if the contribution is the integration and the empirical study. The single clear-day baseline is a genuine data point, just a weak one.\n\nOverall, this would make a useful workshop-level or short-conference paper after the authors add repeated trials, report baselines for all conditions, and separate the optimization set from an honest held-out evaluation. The idea of using closed-loop landing success to select data-augmentation parameters is worth getting in front of the community.\n\nI'd send it to peer review, because the framework is coherent and the gap between claim and evidence is fixable. A serious referee can ask for the missing numbers. I would not cite it myself until the statistics are cleaned up.","headline":"Sensible BO-for-augmentation integration for VTOL landing, but the 20% improvement claim rests on a single 10-trial comparison and no baselines for night conditions.","tokens_in":14889,"tokens_out":2658,"would_cite":false,"duration_ms":23368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bayesian optimization over two augmentation settings lifts a simulated VTOL's landing success from 50% to 70%.","keywords":["Bayesian optimization","data augmentation","VTOL landing","perception DNN","YOLO object detection","CARLA simulator","GUAM vehicle dynamics","sim-to-real gap"],"falsifier":"Repeat the final landing evaluation with the paper's augmentation pair (scale=0.77, brightness=0.66) and with a baseline detector over a larger batch, for example 100 randomized starting positions per scenario; if the 95% confidence intervals for landing success overlap, the reported improvement is within trial noise. A real-flight test with the same camera and controller that shows no improvement would likewise falsify the transfer claim.","tokens_in":13799,"feed_emoji":"🚁","tokens_out":8486,"duration_ms":67527,"temperature":0.7,"pith_summary":"This paper claims that the perception detector guiding a VTOL landing can be tuned substantially without collecting new real-world data, by closing a loop between simulation and retraining. The detector is retrained with image augmentation parameters—scale and brightness—selected by Bayesian optimization, where the objective is the landing success rate measured in a photorealistic urban simulator coupled to a high-fidelity VTOL dynamics model. The authors report that after 30 optimization iterations the simulated success rate rises from 50% to 70%, and that a single parameter pair (scale 0.77, brightness 0.66) keeps the detector working across clear day, clear night, and night-with-rain scenarios. If these simulation results transfer to physical flight, the framework would let perception be adapted to safety-critical landing maneuvers without the cost of exhaustive real-world data collection.","feed_headline":"Bayesian tuning lifts VTOL landing success from 50% to 70%","feed_subtitle":"Retrained YOLO on Bayesian-chosen scale and brightness lands more reliably across day, night, and rain in simulation.","key_machinery":"The load-bearing mechanism is the closed retraining loop. A Gaussian Process models the unknown mapping from augmentation parameters (scale, brightness) to landing success rate, and an Upper Confidence Bound acquisition function proposes the next parameter pair to test; the detector is retrained with those parameters and the success rate from ten simulated landings updates the surrogate. This sample-efficient search is what makes 30 retraining iterations feasible, and the resulting surrogate contour maps identify scale=0.77 and brightness=0.66 as a common high-performing region across the three scenarios.","core_discovery":"The central claim is that repeatedly retraining a helipad detector with data augmentations chosen by a Gaussian-process surrogate—using the simulated landing success rate as the reward—discovers augmentation settings that generalize across lighting and weather. In the paper's experiments, the initial detector trained with default random augmentations reached 50% landing success in clear-day conditions; the Bayesian loop improved that to 70% in both clear day and clear night, kept 50% in night rain, and raised the detector's confidence at extreme start offsets from 30% to 50% in clear day and from 0% to 50% in clear night. The authors also report that optimizing only under clear day produced parameters that failed completely at night, whereas optimizing under adverse conditions yielded a parameter pair shared by all three scenarios.","pith_inferences":["Editorial extension: the ten-trial success metric has 10-percentage-point resolution, so the reported 50%-to-70% gain is two extra successful landings; repeating the final evaluation over 100 or more starting positions and comparing distributions would show whether the GP was fitting signal or trial noise.","Editorial extension: the decisive test is a sim-to-real transfer experiment with the same downward-facing camera, tracking, and PID controller on a physical VTOL, checking whether the simulation-optimized augmentation pair retains its advantage.","Editorial extension: the two-parameter search could be widened to other augmentation dimensions such as hue, rotation, blur, or occlusion; the Bayesian loop's sample efficiency would limit how many dimensions can be explored at the same cost."],"forward_implications":["If the numbers hold, one detector retrained with scale=0.77 and brightness=0.66 can be used without per-condition retraining in clear day, clear night, and night rain.","The night-rain success rate stays at 50% for the common model, so the framework's gain is not uniform; adverse conditions remain the hardest scenario.","Because the objective is measured in simulation, the same loop can be rerun for a new aircraft, camera placement, or landing-pad design before any hardware exists.","The clear-day-only optimization failure at night indicates that augmentation parameters should be tuned under the target operating conditions, not only under nominal ones."],"supporting_citations":[{"why":"Supplies the YOLO detector family used as the real-time object detection backbone for helipad detection.","marker":"[3]"},{"why":"Supplies the photorealistic urban simulation environment that provides the landing scenes and weather conditions.","marker":"[13]"},{"why":"Supplies the high-fidelity 6-DOF VTOL vehicle dynamics model used in the landing simulation.","marker":"[14]"},{"why":"Provides the Python/JAX implementation of the GUAM dynamics that runs at up to 200 Hz in the control loop.","marker":"[38]"},{"why":"Provides the Google Earth helipad image dataset used to train the initial detector.","marker":"[29]"},{"why":"Grounds the choice of Bayesian optimization for optimizing an unknown objective function.","marker":"[22]"},{"why":"Supplies the practical Bayesian optimization method used to select the next hyperparameter pair.","marker":"[23]"},{"why":"Provides the Python Gaussian-process/UCB implementation used in the Bayesian optimization algorithm.","marker":"[45]"},{"why":"Provides the tracking method that keeps the correct helipad bounding box as the source of the control error.","marker":"[41]"},{"why":"Supplies the PID control design that converts detection error into corrective velocity.","marker":"[42]"}],"fun_headline_variants":["Bayesian tuning raises VTOL landing success from 50% to 70%","Bayesian-selected augmentations improve landings across day, night, and rain","GP-tuned data aug boosts drone perception for safer landings","How Bayesian optimization helps VTOLs land reliably in simulation","Bayesian data augmentation sharpens helipad detection for drones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire reported gain rests on the assumption that the simulated landing success rate—ten trials with a PID controller in a photorealistic urban simulator—is a faithful, low-noise proxy for how the same detector would perform on a physical VTOL, and the paper provides no repeated trials, variance estimate, or real-flight validation for that proxy.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian tuning raises VTOL landing success from 50% to 70%","Bayesian-selected augmentations improve landings across day, night, and rain","GP-tuned data aug boosts drone perception for safer landings","How Bayesian optimization helps VTOLs land reliably in simulation","Bayesian data augmentation sharpens helipad detection for drones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001732,"raw_usage":{"total_tokens":6871,"prompt_tokens":992,"completion_tokens":5879,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":5786}},"tokens_in":608,"tokens_out":5879,"duration_ms":40971,"temperature":1.0,"reasoning_tokens":5786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:37:59.637547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the final landing evaluation with the paper's augmentation pair (scale=0.77, brightness=0.66) and with a baseline detector over a larger batch, for example 100 randomized starting positions per scenario; if the 95% confidence intervals for landing success overlap, the reported improvement is within trial noise. A real-flight test with the same camera and controller that shows no improvement would likewise falsify the transfer claim.","supporting_citations":[{"cited_title":"YOLO9000: better, faster, stronger,","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLO detector family used as the real-time object detection backbone for helipad detection."},{"cited_title":"CARLA: An Open Urban Driving Simulator,","cited_arxiv_id":null,"evidence_quote":"Supplies the photorealistic urban simulation environment that provides the landing scenes and weather conditions."},{"cited_title":"Generic Urban Air Mobility (GUAM),","cited_arxiv_id":null,"evidence_quote":"Supplies the high-fidelity 6-DOF VTOL vehicle dynamics model used in the landing simulation."},{"cited_title":"JAX-GUAM: General Uncertainty Approximation Models,","cited_arxiv_id":null,"evidence_quote":"Provides the Python/JAX implementation of the GUAM dynamics that runs at up to 200 Hz in the control loop."},{"cited_title":"HelipadCat: Categorised Helipad Image Dataset and Detection Method,","cited_arxiv_id":null,"evidence_quote":"Provides the Google Earth helipad image dataset used to train the initial detector."},{"cited_title":"Taking the human out of the loop: A review of Bayesian optimization,","cited_arxiv_id":null,"evidence_quote":"Grounds the choice of Bayesian optimization for optimizing an unknown objective function."},{"cited_title":"Practical bayesian optimization of machine learning algorithms,","cited_arxiv_id":null,"evidence_quote":"Supplies the practical Bayesian optimization method used to select the next hyperparameter pair."},{"cited_title":"Bayesian Optimization: Open source constrained global optimization tool for Python,","cited_arxiv_id":null,"evidence_quote":"Provides the Python Gaussian-process/UCB implementation used in the Bayesian optimization algorithm."},{"cited_title":"Simple online and realtime tracking,","cited_arxiv_id":null,"evidence_quote":"Provides the tracking method that keeps the correct helipad bounding box as the source of the control error."},{"cited_title":"J., and Murray, R.,Feedback systems: an introduction for scientists and engineers, Princeton university press, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the PID control design that converts detection error into corrective velocity."}],"review_version":1}