{"id":"69837aa9-7e72-45bf-9416-563ca1e7cc71","arxiv_id":"2608.09467","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RecoverFly uses failure replay, a scene curriculum, and KL regularization in a PPO-based post-training stage to raise UAV vision-language navigation success rates across seen and unseen environments.","lead":"Aerial robots that follow spoken instructions often fail in closed-loop flight because they only learn from expert demonstrations. This paper shows that retraining a drone navigation model with reinforcement learning that replays its own past failures improves success rates by 3 to 8 percentage points on a standard benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed gains over AerialVLA may rest on unreproduced baseline numbers, since Tables 1-3 quote published results rather than rerunning the same checkpoint in the authors' evaluation pipeline.","rationale":"The paper's stated contribution is an empirical framework; the strongest claim is the quantitative improvement over AerialVLA on TravelUAV. For that claim to hold, the comparison must be apples-to-apples. The paper's own sentence 'we report results directly from the corresponding publications' means the comparison is not controlled by the authors. Published baselines can differ from a local rerun due to evaluation-script version, random seeds, physics timestep, success-radius tolerance, or even the definition of a terminal 'away' event. The TravelUAV benchmark is public, and the AerialVLA checkpoint is in hand (it initializes RecoverFly), so a local baseline run is both feasible and necessary. Without it, the 3.12-8.37 point improvement is a comparison of one laboratory's measurements to another's, which is a classic source of inflated gains. This concern is more load-bearing than the reader's token-level credit assignment: even if the shared-action-advantage assumption in Eq. 4 is imperfect, PPO still optimizes a valid (if biased) surrogate, and the empirical result could still hold; the baseline-comparability problem, if real, invalidates the empirical result directly. The missing appendices compound the issue by leaving the exact Stage II sampling and hyperparameters unverifiable. The paper is promising and the ablations are internally consistent, but the central claim should not be accepted at face value until the AerialVLA baseline is reproduced. This does not change the reader's CONDITIONAL verdict, so the recommendation is UNCHANGED.","tokens_in":13053,"tokens_out":9543,"duration_ms":85304,"concrete_test":"Run the exact AerialVLA checkpoint used to initialize RecoverFly through the authors' own evaluation script on the Seen (1,418), Unseen Map (958), and Unseen Object (629) splits, with the same number of evaluation seeds as RecoverFly (three) and the same environment binary and metric code. Compare the obtained SR, NE, OSR, and SPL with the published AerialVLA values quoted in Tables 1-3. If the rerun values lie within the reported 3.12-8.37 pp gain (i.e., if the gap between rerun AerialVLA and RecoverFly shrinks or reverses), the headline claim is not established; if the rerun matches the quoted numbers, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—RecoverFly improves AerialVLA by 3.12 to 8.37 SR points and is best on all three TravelUAV splits—depends directly on the AerialVLA numbers used as the comparison. In the Baselines paragraph the paper states: 'For all baselines, we report results directly from the corresponding publications on the same evaluation splits.' This means the AerialVLA rows in Tables 1-3 are quoted from the AerialVLA paper, not produced by running the AerialVLA checkpoint through the exact evaluation pipeline used for RecoverFly (same AirSim version, TravelUAV environment, metric implementation, success thresholds, termination rules, and seeds). Even small differences in these details can shift SR by several points. Since AerialVLA is the initialization and the entire paper frames the comparison as 'a direct evaluation of RL post-training,' the failure to rerun this baseline is a gap between claim and evidence. The reader's identified weakest assumption (token-level credit assignment in Eq. 4) is a methodological concern that would affect interpretation, but it does not threaten the empirical outcome the way an incomparable baseline does. The ablations in Table 4 inherit the same vulnerability because they also reference the quoted AerialVLA value. The paper also defers key details (rare-scene partition, Stage II sampling, hyperparameters, replay statistics) to Appendices A.1, A.2, B, and C, which are not present in the submitted text, preventing independent verification of the setup.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents RecoverFly, a reinforcement-learning (RL) post-training framework for end-to-end UAV vision-language-action (VLA) policies on the TravelUAV benchmark. The method initializes from the AerialVLA checkpoint (an OpenVLA-7B backbone with a LoRA adapter) and combines four components: token-level PPO with a shared, action-level advantage for grammar-constrained autoregressive action tokens (Eq. 4), a dynamic failure replay pool that re-samples unresolved task initializations under the current policy (Eq. 5), a two-stage long-tail scene curriculum that moves from empirical scene frequencies to equal-quota sampling (Eq. 6), and stage-wise reference-policy KL regularization (Eq. 7). With a rollout budget of about 30% of the training set, the paper reports success-rate improvements over AerialVLA of 8.37, 5.39, and 3.12 percentage points on the Seen, Unseen Map, and Unseen Object splits, respectively, and best overall SR and SPL on all three splits, with three-seed means and standard deviations. Incremental ablations (Table 4) attribute the gains to the combination of failure replay, KL regularization, and the two-stage curriculum.","tokens_in":13438,"tokens_out":15274,"duration_ms":120202,"significance":"If the reported results hold, the paper makes a solid empirical contribution: it demonstrates that closed-loop RL post-training of a 7B-scale VLA policy is feasible at a modest rollout budget (about 30% of the training split) and yields practically meaningful gains (3 to 8 SR points) that are consistent across seen, unseen-map, and unseen-object splits. Strengths of the submission include the reporting of three-seed means and standard deviations for the main results, a clean ablation design that isolates each component, evaluation on held-out splits with no circularity between the fitted policy and the reported numbers, and transparent reporting of the compute budget (21 hours on 8 A100 GPUs). The consistently larger gains on hard trajectories than on easy ones are a credible sign that corrective feedback targets the error-accumulation regime. The main reservations concern the quoted (not re-run) AerialVLA baseline that anchors the headline claim, the absence of the appendices that contain the experimental configuration, and the single-seed ablation protocol.","major_comments":[{"comment":"The AerialVLA rows in Tables 1–3 are quoted from the AerialVLA publication rather than produced by re-running the AerialVLA checkpoint in the authors' evaluation harness. The 'Baselines' paragraph states: 'For all baselines, we report results directly from the corresponding publications on the same evaluation splits,' which is in tension with the paper's characterization of the AerialVLA comparison as 'a direct evaluation of RL post-training.' Because RecoverFly is initialized from AerialVLA, the authors necessarily have the checkpoint and can re-run it under their exact evaluation conditions (AirSim/TravelUAV version, metric implementation, success thresholds, termination rules, seeds). This is load-bearing for the headline claim: the gains of 8.37, 5.39, and 3.12 SR points are differences against these quoted values, and on the Unseen Object split the 3.12-point margin is modest relative to both typical cross-harness variability and the reported seed variance of RecoverFly itself (std 0.81), while the baseline's own variance is unreported. Table 4 compounds the issue because every ablation delta is computed against the same quoted AerialVLA values. I would accept quoted published numbers for third-party baselines, but for the initialization policy the comparison should be measured in the same harness; please re-run AerialVLA (and, if feasible, at least the strongest other baselines) and report the measured values.","section":"Experiments, 'Baselines' paragraph; Tables 1–3"},{"comment":"Several load-bearing configuration details are deferred to appendices that are absent from the submitted text: the reward constants (r_prog min/max and κ_p in Eq. 2), the replay ratio η and the maximum replay attempts N_max in Eq. 5, the KL and value coefficients β and c_v in Eq. 8, all PPO/GAE hyperparameters (γ, λ, ε, learning rate), and the details of the rare-scene partition and Stage II sampling strategy (Appendix A.1), the remaining implementation configuration (Appendix A.2), the replay behavior analysis (Appendix B), and the token-level versus sequence-level PPO comparison (Appendix C). The paper itself states that 'the details about the rest configurations can be found in Appendix A.2,' and that Appendix A.1 contains the Stage II sampling details, but no such material is available in the version under review. Without it, Eqs. 2 and 8 cannot be instantiated by an independent reader, and the paper's claim that token-level PPO provides stable optimization cannot be checked. Please include the appendices (or move the essential hyperparameters and sampling details into the main text).","section":"Method (Eqs. 2, 8); Experiments, 'Implementation Details'; Appendices A.1, A.2, B, C"},{"comment":"Eq. 4 assigns the same action-level advantage Â_t to every valid token of a decoded action, presupposing that credit for a delayed navigation outcome is uniform across tokens with different semantic roles: the three control tokens ⟨Δx, Δz, Δψ⟩ and the optional LAND token. Success or failure depends much more directly on the LAND decision than on, say, a small altitude adjustment, so if per-token credit is not uniform, the per-token clipped ratio in Eq. 4 can bias the update toward tokens that were merely unlikely rather than causally responsible. I regard this as a correctness-risk concern rather than an observed failure; however, the paper's stated comparison of token-level versus sequence-level PPO is deferred to the absent Appendix C, so the choice of backbone is currently unverified. Please report that comparison, add an analysis of per-token credit allocation (for example, ablating shared versus token-specific advantages), or justify the uniform-credit assumption from the action grammar.","section":"Method, 'Token-Level Policy Optimization Backbone' (Eq. 4)"},{"comment":"All ablations in Tables 4 and 5 use seed 1 only, while Tables 1–3 report three-seed standard deviations. Under those variances, several ablation deltas are comparable to the noise level: ID1 versus AerialVLA on Seen (+0.21; std 0.15), ID3 versus AerialVLA on Unseen Map (−0.11; std 1.01), and the Table 5 rows carry no variance at all. In particular, the −6.37-point Unseen Map change when adding failure replay alone (ID2) is a single draw; the interpretation that replay 'redistributes performance across splits rather than providing a uniform gain' needs multi-seed support. Please report at least three-seed means and standard deviations for the ablation conditions, or justify explicitly why single-seed results are sufficient here.","section":"Experiments, 'Ablation Study' (Tables 4–5)"}],"minor_comments":[{"comment":"Subscript and superscript formatting is inconsistent and frequently missing: for example, Eq. (2) renders 'κp(Dt−1−Dt)', Eq. (3) uses 'dt' and 'δt' without subscripts, and 'bAt' appears in place of Â_t in the token-level objective. Please correct the typesetting throughout.","section":"Throughout (Eqs. 2–8)"},{"comment":"The AerialVLA baseline row shows four dashes ('− − − −') although the table has only three framework-component columns, and the rows would benefit from explicit labels ('AerialVLA', 'ID1', etc.). Please clarify the header and row structure.","section":"Table 4"},{"comment":"There are several typos: 'AsillustratedinTable4' (missing spaces), 'Furthermore, We isolate' (mid-sentence capitalization), and 'The state σ_i ... records the active state' should read 'records whether the entry is active, solved, or dropped.'","section":"Experiments, 'Ablation Study'"},{"comment":"In the sentence 'A sequence-level alternative forms a joint ratio ... while this applies a single importance weight,' the referent of 'this' is ambiguous; name the sequence-level objective explicitly.","section":"Method, discussion of Eq. (4)"},{"comment":"The rollout budget of 'approximately 30% of the training-set size' should specify whether replay rollouts are counted in that budget and how the budget is split between Stage I and Stage II; as written, the budget accounting is ambiguous.","section":"Experiments, 'Implementation Details'"},{"comment":"The sentence 'RecoverFly outperforms all comparison methods across all three splits' is stronger than the tables support on every metric (e.g., LongFly achieves higher OSR on the Seen Full set and lower NE and higher OSR on the Unseen Object Hard subset); consider qualifying the claim to SR and SPL.","section":"Conclusions"}],"recommendation":"major_revision","confidential_remarks":"To the editor: I recommend that the submitted version include the referenced appendices before acceptance is considered, and that the handling editor ask the authors to clarify the novelty boundary with respect to RLinf, since the overview states that RecoverFly 'builds on the native token-level VLA optimization in RLinf' while the contributions list presents token-level RL as a contribution; the delta beyond integration should be explicit. The paper's scope fits an applied embodied-AI venue, and the empirical protocol (held-out splits, three-seed main results) is otherwise in good shape."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful empirical paper that deserves peer review, but the headline comparison to AerialVLA is not yet trustworthy because the baseline numbers are quoted from the original paper rather than rerun in the authors' own evaluation pipeline.\n\nWhat is actually new is the specific combination of known ingredients — token-level PPO from RLinf, failure replay reminiscent of prioritized level replay, a two-stage long-tail curriculum, and KL regularization — applied to end-to-end UAV-VLA post-training. That combination is not in the prior literature, and the measured gains of 3 to 8 SR points across three TravelUAV splits are consistent enough that I believe a real effect exists. The paper is also honest about its components and cites the relevant prior work; it does not oversell the novelty of any single mechanism.\n\nThe main soft spot is exactly what the stress-test note flags: the abstract and tables make a strong claim about beating AerialVLA, but AerialVLA is not rerun. The paper states that all baseline numbers are taken directly from prior publications. For secondary baselines that is common and acceptable, but here AerialVLA is the initialization, so the authors almost certainly have the checkpoint. Not evaluating it under the same AirSim version, success thresholds, and metric implementation leaves open the possibility that evaluation differences explain part of the gain. This is the load-bearing comparison and it needs to be fixed.\n\nOther issues are smaller but real: the ablations use only seed 1, the reward and replay hyperparameters are deferred to missing appendices, and no code is released. The reader's worry about token-level credit assignment (Eq. 4) is a legitimate methodological caveat — tokens like LAND vs. altitude change likely deserve different advantage weights — but it is a secondary concern, not a fatal one; the approach inherits from RLinf and the empirical question is whether it works in practice.\n\nWho is this for? People working on UAV-VLN and anyone fine-tuning VLA policies with online RL. The recipe is simple and could be reproduced if the details are released. I would send it to serious review, but only with the requirement that the authors rerun AerialVLA through their own pipeline, add multi-seed ablations, and supply the appendix material. If that revision comes through, the paper becomes a credible, useful contribution to a niche but active area.","headline":"Solid RL post-training recipe for UAV-VLA, but the headline gains over AerialVLA rest on quoted baselines rather than a same-pipeline rerun.","tokens_in":13955,"tokens_out":2671,"would_cite":false,"duration_ms":26674,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RecoverFly shows that re-training a UAV vision-language-action policy with token-level RL, failure replay, and a long-tail curriculum improves navigation success rates by 3.12 to 8.37 percentage points.","keywords":["aerial vision-language navigation","reinforcement learning post-training","token-level PPO","failure replay","long-tail scene curriculum","KL regularization","end-to-end VLA policy","TravelUAV benchmark"],"falsifier":"An ablation that swaps the token-level objective for a sequence-level objective while keeping all other components identical; if success rates do not drop, the token-level credit assignment is not the cause of the gains.","tokens_in":12895,"feed_emoji":"🚁","tokens_out":5818,"duration_ms":38327,"temperature":0.7,"pith_summary":"This paper claims that an end-to-end aerial vision-language-action (VLA) policy can be substantially improved after initial behavior cloning by a closed-loop reinforcement learning post-training stage that deliberately revisits its own failures. On the TravelUAV benchmark, the proposed framework, RecoverFly, combines token-level PPO with a dynamic failure replay pool, a two-stage long-tail scene curriculum, and reference-policy KL regularization. The result is the best reported success rate on the seen, unseen-map, and unseen-object splits, with absolute success-rate gains of 3.12 to 8.37 percentage points over the AerialVLA baseline under a rollout budget of roughly 30% of the training set. The paper argues that the key to these gains is turning sparse, delayed failure signals into reusable corrective experience while keeping the policy from drifting away from its learned capabilities.","feed_headline":"RL post-training lifts UAV navigation success by up to 8.4 points","feed_subtitle":"Failure replay and a rare-scene curriculum turn closed-loop mistakes into corrective training.","key_machinery":"The central mechanism is the token-level PPO objective (Eq. 4), which applies the clipped importance-ratio surrogate separately to each valid action token while sharing a single action-level advantage across the tokens that decode into one 3-DoF command. This avoids a sequence-level joint ratio and per-token clipping decision, allowing the autoregressive action grammar to be optimized with online navigation rewards. Around this backbone, the dynamic failure replay pool (Eq. 5) stores unresolved task initializations and regenerates rollouts with the current policy, and the two-stage scene curriculum plus stage-wise reference-policy KL loss (Eq. 7) jointly steer the training distribution toward long-tailed scenes while constraining policy drift.","core_discovery":"The central discovery is that treating reinforcement learning as a failure-aware post-training stage, rather than a generic second-stage optimizer, converts closed-loop execution mistakes into a reusable learning signal for an autoregressive UAV control policy. RecoverFly stores unresolved task initializations in a dynamic failure pool and regenerates rollouts with the current policy, so that on-policy PPO keeps reliving the most informative failures without ever updating on stale trajectories. Token-level PPO assigns the same action-level advantage to every valid token in the autoregressive action sequence, enabling stable optimization of the grammar-constrained 3-DoF action space. A two-stage curriculum first trains under the original scene distribution and then rebalances toward rare scenes, while stage-wise KL regularization anchors the policy to the initial VLA policy and to the Stage I policy, limiting distribution shift. On TravelUAV, this combination yields the best performance across all three evaluation splits and improves success rate by 3.12 to 8.37 percentage points over the AerialVLA initialization.","pith_inferences":["The same failure-aware replay plus token-level RL recipe could apply to other grammar-constrained autoregressive action spaces, such as grounded manipulation or embodied instruction following, where action tokens have heterogeneous semantic roles.","The token-level advantage sharing is a strong credit-assignment assumption; a natural extension is to weight tokens by their predicted effect on the outcome rather than giving all tokens equal blame.","The reported gains are on a simulated benchmark; a testable extension is whether the learned recovery behavior transfers to real quadcopter flights under domain shift or with added sensor noise.","The curriculum currently acts on scene frequencies; a further step would be to make the curriculum adaptive to the policy's own failure distribution, coupling failure replay with scene difficulty."],"forward_implications":["End-to-end UAV-VLA policies can be improved without extra annotations, external detectors, or hand-designed controllers, purely from closed-loop interaction feedback.","The largest gains occur on hard, long-horizon trajectories, suggesting that corrective RL post-training addresses error accumulation rather than only refining short-range control.","Because the improvement transfers to unseen maps and unseen objects, the learned recovery behavior generalizes beyond the training environments.","The two-stage curriculum shows that deliberately rebalancing scene frequencies during post-training outperforms both uniform sampling and the original distribution, even with the same number of training stages.","A total rollout budget of about 30% of the training-set size is sufficient to achieve these gains, indicating sample-efficient use of online interaction."],"supporting_citations":[{"why":"AerialVLA is the end-to-end UAV-VLA baseline that RecoverFly initializes from and compares against; the improvement over it is the central claim.","marker":"Xu et al. 2026"},{"why":"TravelUAV provides the training and evaluation splits (Seen, Unseen Map, Unseen Object) on which all results are reported.","marker":"Wang et al. 2025"},{"why":"RLinf supplies the token-level VLA optimization backbone that RecoverFly adapts into its PPO implementation.","marker":"Yu et al. 2025"},{"why":"Proximal policy optimization is the underlying RL algorithm whose clipped surrogate is applied at token level.","marker":"Schulman et al. 2017"},{"why":"Generalized advantage estimation is used to propagate delayed event rewards to earlier actions in long-horizon trajectories.","marker":"Schulman et al. 2016"},{"why":"Hindsight experience replay motivates the idea of reusing unsuccessful experience, which the dynamic failure pool builds upon.","marker":"Andrychowicz et al. 2017"},{"why":"Prioritized level replay motivates focusing training on high-learning-value failures, adapted here to scene-level replay.","marker":"Jiang, Grefenstette, and Rocktäschel 2021"},{"why":"OpenVLA-7B is the vision-language-action backbone that, with LoRA, forms the base policy.","marker":"Kim et al. 2025"}],"fun_headline_variants":["RL post-training lifts UAV navigation success by up to 8.4 points","Failure replay and rare-scene curriculum boost UAV navigation RL","RecoverFly turns closed-loop mistakes into UAV navigation wins","Failure-aware token-level RL improves UAV navigation by up to 8.4 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the delayed success or failure of a flight is spread evenly across all the tokens that make up a single control command, so that rewarding each token with the same advantage is a fair credit assignment.","fun_headline_variants_meta":{"raw":{"variants":["RL post-training lifts UAV navigation success by up to 8.4 points","Failure replay and rare-scene curriculum boost UAV navigation RL","RecoverFly turns closed-loop mistakes into UAV navigation wins","Failure-aware token-level RL improves UAV navigation by up to 8.4 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000947,"raw_usage":{"total_tokens":4069,"prompt_tokens":998,"completion_tokens":3071,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":2995}},"tokens_in":614,"tokens_out":3071,"duration_ms":19522,"temperature":1.0,"reasoning_tokens":2995,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:23:11.641035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An ablation that swaps the token-level objective for a sequence-level objective while keeping all other components identical; if success rates do not drop, the token-level credit assignment is not the cause of the gains.","supporting_citations":[{"cited_title":"OpenVLA:","cited_arxiv_id":null,"evidence_quote":"OpenVLA-7B is the vision-language-action backbone that, with LoRA, forms the base policy."}],"review_version":2}