{"id":"24a5ca06-43a6-4d9b-a885-d3950c3988bf","arxiv_id":"2505.12835","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FlightGPT, a 7B vision-language model fine-tuned with supervised demonstrations then GRPO reinforcement learning, reports state-of-the-art success rates on CityNav UAV navigation, including a 9.22 percentage point gain over the strongest baseline in unseen environments.","lead":"The paper builds a drone navigation system, FlightGPT, that reads a city map and a text instruction, reasons out loud, and predicts where to fly. It reports the best scores yet on the CityNav benchmark, especially in never-seen city areas, using a two-stage fine-tune-then-reinforce training recipe.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim depends on an undocumented baseline action-planning protocol; if Qwen2.5-VL-32B or GPT-4o did not receive the same AerialVLN look-ahead controller as FlightGPT, the 9.22-point SR gain may come from the planner rather than the SFT+GRPO pipeline.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation does not document whether baselines use the same action-planning and input protocol as FlightGPT. I agree that this is the single most important threat to the central SOTA claim. The claim is specifically about a 9.22-percentage-point success-rate improvement over the strongest baseline on test-unseen, and the paper's architecture separates target prediction (the trained VLM) from action execution (the AerialVLN look-ahead controller). Since SR measures the full navigation episode, the comparison is only meaningful if all methods share the same execution interface. The paper provides a detailed prompt and system description for FlightGPT, and the ablation study gives internal evidence that SFT+GRPO helps, but external baseline parity is not documented. This is a fixable experimental-protocol issue rather than an internal inconsistency, so the appropriate verdict remains conditional. Secondary issues such as missing error bars and the LLM-based reasoning-quality judge are real but less load-bearing; they would affect confidence intervals and the interpretability subclaim, not the core comparison as directly. Overall, the paper is coherent, discloses limitations in the simulation-to-real gap, and makes a plausible case for the method; a targeted parity experiment would settle the strongest objection.","tokens_in":23414,"tokens_out":4041,"duration_ms":43941,"concrete_test":"Run a controlled parity test on CityNav test-unseen: (1) give Qwen2.5-VL-32B, GPT-4o, and LLaMA-3.2-11B-Vision the exact FlightGPT input (same semantic map rendering and prompt template from Appendix A) and feed their predicted target coordinates into the same AerialVLN look-ahead planner; recompute Table 2. (2) Ablate FlightGPT by replacing the look-ahead planner with a straight-line or shortest-path action policy while keeping the same predicted coordinates. If the VLM baselines reach FlightGPT-level SR with the planner, or if FlightGPT's SR drops substantially without it, the headline gain is not attributable to the trained policy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a system-level comparison, but FlightGPT's system includes more than the trained VLM: Section 3.2.1, step 3, adds the AerialVLN look-ahead mechanism to convert the predicted target coordinate into executable actions. Section 4.1.3 and Appendix B describe the baselines only as Seq2Seq/CMA/MGP-style predictors or as VLMs, without stating whether they receive the same semantic map, the same prompt template, and the same look-ahead controller. CityNav SR is an end-to-end navigation metric, so target-prediction quality and action execution are confounded. If Qwen2.5-VL-32B and GPT-4o were evaluated zero-shot with raw action outputs or without the planner, the reported 9.22-percentage-point test-unseen gap could reflect the action planner rather than the SFT+GRPO training. The internal ablation (SFT-only, RL-only, SFT+RL) plausibly controls for this within FlightGPT, but it does not establish parity with the external baselines. The paper's own limitation section does not mention this evaluation-protocol gap, which is the most load-bearing unverified premise for the SOTA claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FlightGPT, a UAV vision-and-language navigation system built on Qwen2.5-VL-7B. The system takes a semantic map and a natural language instruction, performs chain-of-thought reasoning inside <think> tags, and outputs a target coordinate inside <answer> tags, which is then converted into executable actions via the AerialVLN look-ahead mechanism. The training pipeline has two stages: supervised fine-tuning on CoT-annotated demonstrations generated by Qwen2.5-VL-32B, followed by GRPO reinforcement learning with a composite reward combining goal accuracy, landmark IoU, and output format compliance. On CityNav, the paper reports state-of-the-art results, including a test-unseen success rate of 21.20% compared to 11.98% for Qwen2.5-VL-32B, and an ablation showing SFT+RL outperforms SFT-only and RL-only. The paper also evaluates reasoning quality with GPT-4o scores and discusses limitations including the simulation-to-reality gap and deployment feasibility.","tokens_in":23606,"tokens_out":2927,"duration_ms":33149,"significance":"If the reported results hold, the paper makes a useful empirical contribution: it shows that a 7B VLM trained with SFT warm-up and GRPO can outperform larger zero-shot VLMs and prior map-based goal predictors on a city-scale navigation benchmark, and it provides a structured, interpretable CoT output format. The internal ablation is informative and the release of code and data is a concrete strength. The significance of the headline SOTA claim, however, depends on whether all baselines are evaluated under the same input, prompting, and action-planning protocol; the paper does not currently establish that parity, so the central claim is not fully verified.","major_comments":[{"comment":"The external baseline protocol is under-specified. FlightGPT's action pipeline includes the AerialVLN look-ahead mechanism to convert a predicted target coordinate into executable actions, and its prompt includes a semantic map with red landmark masks and current pose. The baseline descriptions in Appendix B do not state whether Seq2Seq, CMA, MGP, GPT-4o, Qwen2.5-VL, and LLaMA-3.2-11B-Vision receive the same semantic map, the same prompt template, and the same look-ahead controller. Because SR and SPL are end-to-end navigation metrics, the 9.22 percentage-point test-unseen gap in Table 2 could partly reflect differences in the action-execution component rather than the proposed SFT+GRPO training. The authors should report the exact evaluation protocol for each baseline and, preferably, run all VLM and MGP baselines through the same planner with the same prompt; without this, the headline SOTA claim is not verified.","section":"§4.1.3, Appendix B, and §3.2.1 step 3"},{"comment":"All main results are reported from a single training run without error bars, multiple seeds, or significance tests. Given that the central claim depends on the gap between FlightGPT (SR 21.20) and the strongest baseline (SR 11.98), and that some val-seen versus test-unseen numbers vary substantially within a single model (e.g., Qwen2.5-VL-7B), the reader cannot distinguish a stable improvement from run-to-run noise. Please report at least three seeds with means and standard deviations or bootstrap confidence intervals, and specify the exact CityNav split construction and the test-unseen evaluation procedure, including how target coordinates and episode termination are handled.","section":"§4.2.1, Table 2"},{"comment":"The limitations section does not acknowledge the evaluation-protocol risk identified above. It lists the simulation-to-reality gap and deployment feasibility, but it does not mention that the external baselines may not have used the same look-ahead planner and prompting scheme as FlightGPT. Since that is the most load-bearing unverified premise for the state-of-the-art claim, the limitation should either be stated explicitly or addressed experimentally before the paper claims SOTA performance.","section":"§6 Limitations"}],"minor_comments":[{"comment":"The SFT data collection is described as 1,872 samples collected and filtered from Qwen2.5-VL-32B outputs, but the filtering criteria in §3.2.2 are stated qualitatively ('abnormal output formats', 'more than 20 meters away'). Please specify the exact numbers discarded at each step and the format-validity check used.","section":"§4.1.4"},{"comment":"The data generation step replaces the predicted target location with the ground truth for retained samples. This can create a mismatch between the generated CoT text and the final answer. Please state whether any analysis was done on the frequency of such replacements and whether the CoT text was also corrected to remain consistent with the ground-truth location.","section":"§3.2.2"},{"comment":"The reasoning quality scores are averages of three GPT-4o evaluations over 5,000 samples, but no variance, inter-rater agreement, or correlation with downstream navigation performance is reported. Please add confidence intervals or a per-sample score distribution, and clarify how the random sample was drawn.","section":"§4.2.3, Table 3"},{"comment":"There is a typo in the action list: 'forwad' should be 'forward'.","section":"§3.1"},{"comment":"The CMA baseline is cited as 'Hu et al.' without a year or complete bibliographic entry; the reference list also appears to contain an incomplete entry for Hu et al. Please provide the full citation.","section":"Appendix B"},{"comment":"The paper states that CityNav covers Birmingham and Cambridge but does not clarify which cities appear in val-seen, val-unseen, and test-unseen splits. A sentence describing the split construction would help readers interpret the generalization claims in Table 2.","section":"§4.1.1"}],"recommendation":"major_revision","confidential_remarks":"The central methodological issue is not circularity—the training signal is tied to ground-truth targets and landmarks, and evaluation is on held-out splits—but rather protocol parity. The paper's strongest empirical claim is a system-level comparison, yet the system includes an external planner that baselines may not share. I would encourage the editor to require the authors to either document the exact per-baseline evaluation protocol or re-run the comparison under identical conditions. The paper is otherwise a reasonable incremental contribution to UAV VLN, and the proposed revision is within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper for one reason: it is a real empirical result, not a vaporware system description. A 7B VLM fine-tuned with SFT then GRPO on a composite goal/IoU/format reward beats much larger zero-shot VLMs and the prior map-based goal predictor on CityNav test-unseen. The internal ablation is clean—SFT-only gets 11.20 SR, RL-only 19.80, SFT+RL 21.20—and the training curves in Figure 3 show SFT warm-up accelerates convergence. The reward design is sensible, and the CoT examples in Appendix C show visibly more structured reasoning after SFT+RL. That is real work and real progress for UAV VLN.\n\nThe soft spot is exactly what the stress test flags: baseline parity. FlightGPT converts its predicted target coordinate into actions through the AerialVLN look-ahead mechanism (Section 3.2.1, step 3). The paper never states whether Seq2Seq, CMA, MGP, GPT-4o, Qwen2.5-VL, or LLaMA-3.2-Vision received the same semantic map, the same prompt template, and the same look-ahead controller. Appendix B is unhelpful: it describes Seq2Seq and CMA as direct action generators, while Section 3.2.1 implies they follow the same predict-then-plan paradigm as FlightGPT. That inconsistency makes the 9.22-point success-rate gain over Qwen2.5-VL-32B genuinely hard to attribute. The internal ablation does not resolve this, because all FlightGPT variants use the same planner; it only isolates the training pipeline, not the system-level comparison.\n\nThis is a documentation gap, not a demonstrated fraud. The OSR and SPL numbers for the VLM baselines are high enough that they were probably doing goal prediction rather than raw action generation, but 'probably' is not good enough for a SOTA claim. The other weaknesses are minor by comparison: no error bars or multiple seeds, an LLM judge for reasoning quality without human validation, and a code-release promise that is inconsistent across the abstract, footnote, and conclusion.\n\nBottom line: this paper deserves a serious referee. The central empirical finding is new and the methodology is reproducible in principle. A referee should require: (1) explicit per-baseline inference protocol, especially the action planner; (2) multiple seeds or confidence intervals; (3) human agreement on the reasoning-quality judge; (4) a clear code/data release statement. If the authors can show baseline parity, this becomes a solid reference point for VLM-based UAV navigation. I would bring it to a reading group and would cite it if the protocol question gets answered.","headline":"FlightGPT's 7B SFT+GRPO model posts a genuinely strong CityNav result, but the paper never documents whether the baselines share its AerialVLN look-ahead planner, so the 9.22-point SOTA margin is not yet fully interpretable.","tokens_in":24277,"tokens_out":4971,"would_cite":true,"duration_ms":49495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FlightGPT claims a 9.22-point success-rate gain over the strongest baseline in unseen city navigation.","keywords":["UAV vision-and-language navigation","vision-language models","group relative policy optimization","chain-of-thought reasoning","supervised fine-tuning","CityNav","generalization to unseen environments","aerial navigation"],"falsifier":"Rerun Qwen2.5-VL-32B and MGP on the CityNav test-unseen split with the exact semantic-map prompt and AerialVLN look-ahead planner used by FlightGPT; if their success rates stay near 11.98% and 6.38%, the SFT+GRPO recipe is responsible for the gain, but if they rise toward 21.20%, the planner, not the training, carries the improvement.","tokens_in":23144,"feed_emoji":"🚁","tokens_out":10877,"duration_ms":96011,"temperature":0.7,"pith_summary":"The paper sets out to show that a 7-billion-parameter vision-language model, fine-tuned in two stages, can outperform much larger zero-shot models and prior map-based predictors on city-scale UAV vision-and-language navigation. FlightGPT first runs supervised fine-tuning on chain-of-thought demonstrations produced by a stronger VLM, then applies group relative policy optimization (GRPO) with rewards for goal accuracy, landmark reasoning, and format compliance. On the unseen test split of CityNav, it reports a success rate of 21.20%, 9.22 percentage points above the strongest baseline, Qwen2.5-VL-32B at 11.98%. If the claim holds, generalization in aerial vision-and-language navigation can be bought with training strategy and structured reasoning rather than raw model scale.","feed_headline":"A 7B drone model beats 32B zero-shot models in unseen cities","feed_subtitle":"Two-stage training and chain-of-thought lift CityNav unseen-success from 11.98% to 21.20%.","key_machinery":"The load-bearing machinery is the two-stage training pipeline on top of a vision-language model, with target-coordinate prediction as the intermediate representation. Stage one is supervised fine-tuning on chain-of-thought demonstrations generated by a stronger VLM, which gives the policy a structured initialization. Stage two is Group Relative Policy Optimization (GRPO), a reinforcement-learning update that compares sampled outputs within a group to estimate advantages, driven by a composite reward: goal accuracy (an exponential decay around the 20-meter success threshold), landmark reasoning (intersection-over-union of predicted and ground-truth landmark boxes), and format compliance (whether <think> and <answer> tags parse). At inference, FlightGPT takes a semantic map annotated with the drone's position and known landmarks plus the natural-language target description, emits a chain-of-thought and a predicted target location, and hands that location to the AerialVLN look-ahead mechanism, which simulates future trajectories to produce executable actions. The look-ahead mechanism is what closes the loop between the VLM's coordinate output and the drone's discrete action space.","core_discovery":"The paper's central claim is that FlightGPT, a framework built on the 7B vision-language model Qwen2.5-VL-7B, achieves the best reported results on the CityNav UAV vision-and-language navigation benchmark by combining supervised fine-tuning with a GRPO reinforcement-learning stage and chain-of-thought reasoning. On the test-unseen split, FlightGPT reports a success rate of 21.20%, navigation error of 76.2 meters, oracle success rate of 35.38%, and SPL of 19.24, compared with 11.98% success rate for Qwen2.5-VL-32B, the strongest baseline, and 6.38% for the map-based goal predictor MGP. The framework first predicts a target coordinate from a semantic map and a natural-language instruction, then uses a look-ahead planning mechanism to convert that coordinate into executable drone actions. The contribution is therefore a training-and-representation recipe: structured reasoning plus reinforcement learning, rather than a larger model, is what the authors claim drives generalization to unseen urban environments.","pith_inferences":["Editorial inference: if the reported gain is mostly attributable to the look-ahead planner, the same SFT+GRPO coordinate-prediction recipe could transfer to other map-based navigation settings by swapping the action-conversion step, since the model itself only needs to output a map coordinate.","Editorial inference: because the SFT stage replaces the teacher's predicted target with the ground-truth location, SFT does not actually teach the model to connect its landmark boxes to its final answer; the RL IoU reward may be the only term enforcing that connection, which could be tested by ablating that reward alone.","Editorial inference: the format reward's success at enforcing structured output suggests a general, parse-free way to inject interpretability into closed-loop vision-language agents, namely rewarding tag compliance rather than hand-designing action decoders."],"forward_implications":["A 7B vision-language model trained with SFT plus GRPO can outperform a 32B zero-shot VLM on unseen city-scale navigation, so model scale is not the only lever for generalization.","The two-stage recipe nearly doubles navigation efficiency on unseen test splits: SPL rises from 10.76 for Qwen2.5-VL-32B to 19.24 for FlightGPT.","Adding the RL stage on top of SFT is what buys out-of-distribution robustness: SFT-only reaches 11.20% success on test-unseen, while SFT+RL reaches 21.20%.","Structured chain-of-thought with landmark bounding boxes and target coordinates makes the navigation decision inspectable, and the paper's automated scoring of 5,000 outputs indicates SFT+RL produces more complete and coherent reasoning than RL alone."],"supporting_citations":[{"why":"Supplies the CityNav dataset and the MGP map-based goal-predictor baseline; its metrics (SR, NE, OSR, SPL) define the evaluation.","marker":"Lee et al., 2024"},{"why":"Supplies the AerialVLN look-ahead mechanism that converts the predicted target coordinate into executable drone actions, closing the loop between the VLM output and the action space.","marker":"Liu et al., 2023"},{"why":"Provides Qwen2.5-VL, the 7B base model that FlightGPT fine-tunes and the 32B model used both for SFT data generation and as the strongest zero-shot baseline.","marker":"Bai et al., 2025"},{"why":"Motivates the GRPO reinforcement-learning stage for improving chain-of-thought reasoning and generalization in language and vision-language models.","marker":"DeepSeek-AI et al., 2025"},{"why":"Provides the vision-language reinforcement-learning training implementation used for the GRPO stage.","marker":"Shen et al., 2025"},{"why":"Provides the fine-tuning implementation used for the SFT stage.","marker":"Zheng et al., 2024"}],"fun_headline_variants":["7B drone model beats 32B in unseen cities","FlightGPT: 7B beats 32B in UAV navigation","Reinforcement learning lifts 7B drone nav past 32B","Chain-of-thought helps 7B UAV model beat 32B","Smaller drone model navigates unseen cities better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that all baselines were evaluated with the same input and action-planning protocol as FlightGPT, including the look-ahead mechanism that converts a predicted coordinate into executable actions; the paper does not verify this for the zero-shot VLM baselines.","fun_headline_variants_meta":{"raw":{"variants":["7B drone model beats 32B in unseen cities","FlightGPT: 7B beats 32B in UAV navigation","Reinforcement learning lifts 7B drone nav past 32B","Chain-of-thought helps 7B UAV model beat 32B","Smaller drone model navigates unseen cities better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1770,"prompt_tokens":967,"completion_tokens":803,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":716}},"tokens_in":583,"tokens_out":803,"duration_ms":7842,"temperature":1.0,"reasoning_tokens":716,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:54.112488+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun Qwen2.5-VL-32B and MGP on the CityNav test-unseen split with the exact semantic-map prompt and AerialVLN look-ahead planner used by FlightGPT; if their success rates stay near 11.98% and 6.38%, the SFT+GRPO recipe is responsible for the gain, but if they rise toward 21.20%, the planner, not the training, carries the improvement.","supporting_citations":[],"review_version":1}