{"id":"798acbdd-7885-4508-bea8-bb70d1609835","arxiv_id":"2607.06964","paper_version":1,"verdict":"CONDITIONAL","confidence":"UNKNOWN","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"FRAMe combines an LLM planner with RAG-based memory and a multi-modal coach agent to generate valid, preference-aligned eVTOL flight plans, achieving up to 93.8% validity across four LLMs.","lead":"The paper builds FRAMe, a system that uses large language models to generate eVTOL flight plans from natural language instructions, augmented with retrieval of past plans and a vision-based coach agent that checks validity and preference alignment. A smart generalist might read it to understand how LLMs can be applied to safety-critical planning tasks where human preferences matter.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Preference-capture claim is overgeneralized: o3-mini clearance declines 23% under the clearance preference despite clear headroom, and no error bars make significance unassessable. The reader's LLM-as-judge concern is partially misplaced — preference metrics are computed geometrically, not by the LL","rationale":"The reader's verdict of CONDITIONAL is appropriate, but for a somewhat different reason than stated. The reader's concern about LLM-as-judge bias is a real limitation of the system, but it does not directly undermine the reported preference-capture results because those results are evaluated with geometric metrics, not the LLM judge. The LLM judge affects the system's internal operation (what gets stored in RAG) but not the evaluation.\n\nThe more pressing issue is that the preference-capture claim — the second half of the central claim — is overgeneralized and statistically untested. The paper's own data shows a direct contradiction: o3-mini's clearance declines 23% under the clearance preference despite clear headroom. The 'where the metric has headroom' qualifier doesn't save the claim because this case has headroom and still fails. The paper acknowledges this but treats it as an incidental observation rather than evidence against the general claim.\n\nThe validity claim (first half) is more secure: it's based on geometric checks and shows consistent improvement across all models. However, even here, the absence of error bars means we can't assess whether the differences between conditions (e.g., o3-mini: 79.6% → 79.1% → 88.9%) are statistically significant.\n\nThe paper has real strengths: the warmup/ablation protocol is well-designed, the geometric evaluation framework is sound, the code is publicly available, and the systematic multi-model ablation is valuable. The validity results are likely robust. But the preference-capture claim needs qualification (it's model- and preference-specific, not general) and statistical support.\n\nVerdict recommendation: UNCHANGED. The reader's CONDITIONAL verdict is appropriate, though the reasoning should shift from LLM-as-judge bias to the mixed preference-capture results and lack of statistical testing. The validity claim stands; the preference-capture claim is weaker than presented.","tokens_in":19297,"tokens_out":5376,"duration_ms":292250,"concrete_test":"Re-run the full ablation experiment for o3-mini and o4-mini with at least 5 independent seeds per condition (varying only the LLM sampling seed), and report mean ± standard deviation for all three preference metrics. Specifically check: (1) whether o3-mini's clearance decline under the clearance preference (5.88→4.52 km) is statistically significant — if so, the central claim must be qualified as model-specific rather than general; (2) whether o4-mini's clearance improvement (5.58→6.07 km) exceeds the noise floor — if not, the strongest positive preference-capture result is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader identifies LLM-as-judge bias as the weakest assumption, but this concern is partially misplaced: the preference-capture evaluation in Figure 5 and Table 1 uses geometrically computed metrics (distance, waypoint count, polygon clearance), not the LLM judge. The o4-mini vision agent operates within the system pipeline (coach feedback stored in RAG), but the reported preference-capture results do not depend on its verdicts.\n\nThe more load-bearing concern is that the preference-capture claim is overgeneralized and unsupported by statistical evidence. The central claim states the full system 'shifts preference-relevant metrics in the operator-favored direction where the metric has headroom.' But the paper's own data contradicts this for o3-mini under the clearance preference: clearance declines monotonically from 5.88 km (Baseline) to 5.35 km (+RAG) to 4.52 km (+RAG+Coach) — a 23% decrease. This is a case with unambiguous headroom (A* achieves only 1.91 km, so 5.88 km is far from any ceiling). The paper acknowledges this decline but frames it as 'coach-driven revisions appear to prioritize correcting validity violations over maximizing polygon separation' (Section 3.2), which is an admission that the system fails to capture the stated preference for this model.\n\nAdditionally, no error bars, confidence intervals, or statistical tests are reported anywhere. LLM outputs are stochastic; without repeated runs, the observed movements (e.g., o4-mini clearance +9%, o3-mini waypoints -12%) could be sampling noise. The distance preference shows no movement for any reasoning model, and waypoint improvements are limited to models starting above their floor. The net evidence for preference capture is: one preference shows no effect, one model-preference combination moves in the wrong direction despite headroom, and the remaining positive signals are untested for statistical significance.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper presents FRAMe, an end-to-end LLM-based flight planning system for eVTOL aircraft that integrates a planner LLM with a RAG-based memory module and a multi-modal coach agent. The coach agent performs geometric validity checks and preference-alignment assessment using vision capabilities. The system is evaluated across four LLMs (o3-mini, o4-mini, GPT-5.4, DeepSeek-R1), three difficulty levels, and four conditions (A*, Baseline, +RAG, +RAG+Coach) using a warmup/ablation protocol. The central claims are: (1) the full system yields the highest validity for every planner tested, and (2) the full system shifts preference-relevant metrics in the operator-favored direction where headroom exists. Evaluation uses geometrically computed metrics (distance, waypoint count, polygon clearance) rather than human annotation.","tokens_in":20248,"tokens_out":1098,"duration_ms":276569,"significance":"The paper makes a solid contribution to LLM-based planning in the aviation domain. Strengths include: (1) a systematic ablation protocol with a warmup phase that seeds the RAG database followed by a read-only evaluation phase, cleanly isolating RAG and coach contributions; (2) an annotator-free preference evaluation framework using three geometrically computed metrics, avoiding LLM-as-judge for the reported preference-capture results; (3) evaluation across four state-of-the-art LLMs and three difficulty levels with a classical A* baseline for comparison; (4) reproducible code released at a public repository. The finding that the coach is load-bearing on top of RAG (particularly for o3-mini, where RAG alone regresses validity) is a useful empirical result for the community.","major_comments":[{"comment":"Abstract and Section 3.2: The central preference-capture claim states the full system 'shifts preference-relevant metrics in the operator-favored direction where the metric has headroom.' However, the paper's own data in Figure 5 shows that o3-mini under the clearance preference exhibits a monotonic decline from 5.88 km (Baseline) to 5.35 km (+RAG) to 4.52 km (+RAG+Coach)—a 23% decrease in the wrong direction. A* achieves 1.91 km, so 5.88 km is far from any ceiling, meaning headroom clearly exists. The paper acknowledges this decline but frames it as the coach 'prioritizing correcting validity violations over maximizing polygon separation.' This is an admission that the system fails to capture the stated preference for this model. The abstract's unconditional claim ('shifts preference-relevant metrics in the operator-favored direction where the metric has headroom') is contradicted by o3","section":null}],"minor_comments":[{"comment":"Section 2.3: The RAG retrieval restricts candidates to prior records with identical scenario geometry. This is a strong restriction that limits the generality of the memory system. Please discuss whether this design choice limits applicability to novel scenarios and acknowledge it as a limitation, perhaps in Section 4.","section":null},{"comment":"Section 3.1: The number of scenarios per difficulty level and the total number of evaluation runs per condition are not clearly stated. Please specify the exact scenario counts and the number of runs per cell in the evaluation grid.","section":null},{"comment":"Algorithm 1, line 9: The notation EmbedAll(cands) suggests embedding all candidate preference texts, but the text in Section 2.3 describes embedding only the operator preference. Please clarify the relationship.","section":null},{"comment":"Section 2.2: The coach agent uses o4-mini for preference alignment, but it is unclear whether the same o4-mini model is also used as a planner in the experiments. If so, this creates a self-evaluation dynamic that should be discussed, at least in the limitations.","section":null},{"comment":"Figure 5: The y-axis ranges are narrow, which visually exaggerates small differences. Consider widening the ranges or annotating the percent change from Baseline.","section":null},{"comment":"Appendix D, Figure 9: The per-difficulty breakdown is informative. Consider moving at least a summary of this analysis to the main text, as it shows where the coach and RAG contribute most.","section":null},{"comment":"References: The citation for 'ichter et al., 2023' (SayCan) appears to have a lowercase initial. Please verify.","section":null},{"comment":"Section 1: The term 'GPT-5.4' is used; please verify this is the correct model name and cite it more precisely.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about overgeneralization of the preference-capture claim is well-founded and is the primary reason for the major_revision recommendation. The reader's LLM-as-judge concern is partially misplaced: the preference-capture metrics in Figure 5 and Table 1 are computed geometrically, not by the LLM judge. The LLM judge (o4-mini vision) operates within the system pipeline but does not directly produce the reported preference metrics. The more serious issue is the absence of any statistical evidence (no error bars, no repeated runs) combined with the o3-mini clearance counterexample. The paper would benefit from either softening the preference-capture claim to acknowledge model-specific failures or providing statistical evidence that the positive cases are significant."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful reading and agree the abstract overstates the preference-capture claim. We will revise.","responses":[{"response":"The referee is correct. The abstract's claim is stated unconditionally and is directly contradicted by the o3-mini clearance data in Figure 5. We acknowledge this as a genuine overstatement in the abstract that we will correct. Specifically, we will revise the abstract to read that the full system 'shifts preference-relevant metrics in the operator-favored direction where the metric has headroom for most model–preference combinations,' and will add an explicit caveat noting the o3-mini clearance exception. The body text (Section 3.2) already discusses this case honestly, noting that 'coach-driven revisions appear to prioritize correcting validity violations over maximizing polygon separation,' but the abstract must reflect this qualification as well. We agree this is not merely a rounding or framing issue: o3-mini's clearance under the clearance preference moves 23% in the wrong direction with substantial headroom, and the abstract should not claim otherwise.","revision_made":"yes","referee_comment":"The abstract's unconditional claim that the full system 'shifts preference-relevant metrics in the operator-favored direction where the metric has headroom' is contradicted by o3-mini under the clearance preference, which declines monotonically from 5.88 km (Baseline) to 4.52 km (+RAG+Coach)—a 23% decrease in the wrong direction, with clear headroom (A* achieves 1.91 km). The paper acknowledges this but the abstract does not qualify the claim."}],"tokens_in":18947,"tokens_out":530,"duration_ms":68946,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Two things matter here. First, the validity claim holds up: the full FRAMe system (RAG + coach) achieves the highest validity for every planner tested, and the ablation cleanly isolates that the coach is load-bearing on top of retrieval — most clearly for o3-mini, where RAG alone slightly regresses validity and the coach recovers it. Second, the preference-capture claim is overgeneralized. The paper says the system shifts preference metrics in the operator-favored direction where headroom exists, but o3-mini's clearance declines 23% under the clearance preference despite obvious headroom (A* achieves 1.91 km; o3-mini starts at 5.88 km). The paper acknowledges this but frames it as the coach prioritizing validity over clearance — which is an admission that the system fails to capture the stated preference for that model. The distance preference shows no movement for any reasoning model, which the paper attributes to headroom limits. That explanation is fair for distance, but it doesn't rescue the clearance result for o3-mini. The stress-test note correctly identifies this as the load-bearing weakness. I'd push back on the reader's framing of LLM-as-judge bias as the weakest assumption. The preference-capture metrics in Figure 5 and Table 1 are computed geometrically — distance, waypoint count, polygon clearance — not by the o4-mini vision judge. The judge operates inside the system pipeline (coach feedback stored in RAG), but the reported preference results don't depend on its verdicts. So the judge-bias concern is real for the system's internal operation but doesn't undermine the evaluation. The real problem is statistical: no error bars, no confidence intervals, no repeated runs. LLM outputs are stochastic. The observed movements — o4-mini clearance +9%, o3-mini waypoints -12% — could be noise. With a single run per condition, we can't assess significance. This is the gap a referee should flag. What's genuinely new: the combination of RAG memory with a geometric validity checker and a vision-based coach for eVTOL flight planning. The warmup/ablation protocol is well designed — seeding the database then freezing it for ablation is the right approach. Code is public. The per-preference breakdown in Appendix E is honest about where the system works and where it doesn't. This is a well-engineered system paper with a systematic ablation. The validity results are solid; the preference-capture results are preliminary. It deserves a serious referee who should demand error bars and a more careful statement of the preference-capture claim — particularly the o3-mini clearance regression, which contradicts the headline claim as written.","headline":"Solid system paper on LLM flight planning; preference-capture claim is overgeneralized for one model and lacks error bars","tokens_in":20401,"tokens_out":613,"would_cite":true,"duration_ms":140609,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"LLM flight planner with memory and vision coach hits 94% validity","keywords":["LLM planning","flight planning","eVTOL","retrieval-augmented generation","multi-modal agent","preference alignment","UAV","advanced air mobility"],"falsifier":"If the coach agent's vision-based preference judgments were shown by human raters to disagree systematically with human operator preferences on a representative subset of plans, the preference-capture claims would be undermined regardless of the geometric validity results.","tokens_in":19424,"feed_emoji":"✈️","tokens_out":1166,"duration_ms":165427,"temperature":0.7,"pith_summary":"The paper presents FRAMe, a flight-planning system for eVTOL aircraft built around a large language model that translates natural-language operator preferences into sequences of geographic waypoints. The system pairs the planner LLM with two add-on modules: a retrieval-augmented memory that supplies relevant past flight plans as in-context examples, and a multi-modal coach agent that uses rule-based geometry checks to verify plan validity and a vision-capable LLM to assess whether the route aligns with the operator's stated preference. The central claim is that each component contributes non-redundantly: across four LLMs and three difficulty levels, the full system (RAG plus coach) yields the highest plan validity for every planner tested, reaching up to 93.8% aggregate and 99% on easy scenarios for the strongest planner. The coach is shown to be load-bearing rather than decorative—for one model, retrieval alone slightly regressed validity and the coach recovered the gain. The paper also argues that LLM-based planners are preference-aware where classical algorithms like A* are not: A* achieves perfect validity but is blind to operator intent, producing routes with 3–5× more waypoints and far less clearance from hazards than even the baseline LLM configurations. Preference capture is demonstrated on three measurable objectives—minimize distance, minimize waypoints, maximize clearance—with gains that are model-specific and constrained by how much headroom each planner has at baseline.","feed_headline":"LLM flight planner with memory and vision coach hits 94% validity","feed_subtitle":"Adding retrieval and a multi-modal coach agent to LLM route planning for eVTOL aircraft boosts validity above 93% and shifts routes toward a","key_machinery":"three-stage coach agent (geometric validity + vision-based preference alignment + operator feedback) combined with cosine-similarity RAG retrieval over prior plans for geometrically identical scenarios","core_discovery":"The paper's central result is that combining retrieval-augmented memory with a multi-modal coach agent makes LLM-generated flight plans both more valid and more preference-aligned than either component alone, and that this combination is necessary because retrieval without validation can actively harm validity for certain models. The coach agent's three-stage pipeline—geometric validity checks, vision-based preference assessment, and optional human feedback—creates a feedback loop where evaluated plans are stored in the memory database, progressively improving future retrievals. The ablation across four LLMs isolates each module's contribution: RAG helps most when the planner already follows","pith_inferences":["If the coach's vision-based preference alignment is biased by the judge LLM's priors—as the paper acknowledges—then the preference-capture results may partly reflect the judge's expectations rather than purely objective alignment, which would mean the reported gains are an upper bound on true preference capture.","The system's reliance on exact scenario matching for RAG retrieval (same flyzone, polygons, origin, destination) means the memory only helps when the same geographic problem recurs with different preferences; this limits generalization to novel airspace configurations where no prior plans exist.","The absence of dynamic constraints (traffic, energy limits, vehicle dynamics) means the validity metric measures geometric feasibility only; plans that pass the coach's checks could still be infeasible for real eVTOL operations, so the 93.8% validity rate should not be read as an operational readiness figure.","The monotonic decline in clearance for one model under augmentation suggests a potential tension between the coach's validity-first corrections and preference optimization that could become more pronounced as the number of simultaneous preferences grows."],"forward_implications":["The coach-as-validator pattern could generalize to other LLM planning domains where outputs must satisfy hard geometric or physical constraints—robotics, logistics routing, autonomous driving—provided that rule-based validity checks exist alongside the LLM judge.","The finding that retrieval alone can regress performance without a validation gate suggests that RAG-based planning systems need a filtering or verification step between retrieval and use, particularly for weaker or less instruction-following models.","The modular ablation protocol (warmup-seeded database, then read-only sweep) offers a reusable evaluation methodology for separating the effects of memory, verification, and base model capability in any system that combines LLM planning with retrieval.","The model-specific difficulty profiles (e.g., one model peaking on hard scenarios while another degrades) suggest that planner-model selection for safety-critical applications should be conditioned on the operational complexity envelope, not just aggregate benchmarks."],"fun_headline_variants":["Memory plus vision coach lifts LLM flight plan validity to 94%","RAG and vision coach yield 94% valid LLM flight plans","Coach agent and RAG memory boost LLM flight plan validity","RAG without coach can harm LLM flight plan validity","Vision coach and memory improve LLM eVTOL flight planning"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The preference-alignment assessment relies on a vision-capable LLM judging the planner's output, which inherits the judge model's hidden biases—including a tendency to favor outputs resembling its own style—with no human validation of whether the coach's alignment verdicts actually match what human operators would prefer.","fun_headline_variants_meta":{"raw":{"variants":["Memory plus vision coach lifts LLM flight plan validity to 94%","RAG and vision coach yield 94% valid LLM flight plans","Coach agent and RAG memory boost LLM flight plan validity","RAG without coach can harm LLM flight plan validity","Vision coach and memory improve LLM eVTOL flight planning","Multi-modal coach and RAG push LLM flight plan validity to 94%"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1291,"prompt_tokens":545,"completion_tokens":746,"prompt_tokens_details":null},"tokens_in":545,"tokens_out":746,"duration_ms":49892,"temperature":1.0,"reasoning_tokens":644,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T00:52:55.257939+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the coach agent's vision-based preference judgments were shown by human raters to disagree systematically with human operator preferences on a representative subset of plans, the preference-capture claims would be undermined regardless of the geometric validity results.","supporting_citations":[],"review_version":1}