{"id":"05466670-182a-42d9-be89-a78353d345e8","arxiv_id":"2607.22014","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new 120-mission drone benchmark finds the best off-the-shelf multimodal AI completes 34.8% of missions versus 84.4% for humans, with scaling helping but not closing the gap.","lead":"MissionBench tests 22 multimodal AI models on 120 simulated drone missions, where each model must plan, navigate, and report from a single text instruction. The best model completes about one in three missions, while humans complete about four in five, quantifying the gap between general-purpose AI and reliable drone autonomy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reporting success relies on an LLM judge with a 90% similarity threshold; under exact matching the headline SR halves (34.8->19.1), so the central quantitative claim is threshold-dependent.","rationale":"The reader identified the LLM-as-judge soft matching as the weakest assumption, and my analysis agrees. The exact-match drop (34.8 to 19.1) is reported in the paper itself (Sec. C.6), so this is not speculative — the headline metric is demonstrably sensitive to the evaluation rule. Reporting missions dominate the Test split (17/30), so the effect is first-order. The paper's validation (150 samples, precision 1.0/recall 0.94) is genuine supporting evidence and the judge is more aligned with human semantic judgment than exact match, but the underlying construct — 'success' on a reporting mission — is not pinned down. The abstract says 'succeeds,' which in an embodied benchmark implies the mission's information goal was met; tolerating one or two wrong characters in a license plate number arguably fails that goal. The paper's own numbers show the headline is not threshold-invariant, and the reader's CONDITIONAL verdict is appropriate: the empirical pattern (models far below human, scaling helps) survives the re-scoring, but the specific '34.8%' and the human-model gap need to be reported with the success criterion made explicit and with a strict-criterion variant. I would not move to REJECT because the benchmark construction, closed-loop design, ablations, and honest limitations section provide substantial value and the central difficulty claim is robust across metrics. The single most load-bearing fix is to re-report headline SR under a strict reporting criterion and justify the chosen tolerance per mission type.","tokens_in":23512,"tokens_out":1909,"duration_ms":16228,"concrete_test":"Re-score all 30 Test-split missions using a stricter reporting criterion: (a) exact string match for alphanumeric reports, and separately (b) a stricter LLM-judge prompt requiring 100% character-level agreement for license plates/IDs while keeping semantic tolerance for descriptive attributes. If Gemini 3.1 Pro's SR falls below 25% under (a) or (b), the abstract's 'fewer than 35%' should be re-reported with the criterion stated; if it stays ~34.8%, the concern is resolved. Also report human keyboard-control SR under the same strict criterion to see if the human-model gap narrows.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim ('strongest model succeeds on fewer than 35% of missions vs 84.4% human') rests on the LLM-as-judge soft-matching rule for reporting missions (Sec. C.6, Table C6). Under exact string matching, Gemini 3.1 Pro's SR drops from 34.8% to 19.1% — a 45% relative reduction. Reporting missions are 17/30 (56.7%) of the Test split, so the judge rule directly determines the headline number. The judge prompt instructs that 'if one or two numbers/letters differ, it can still be considered success' — i.e., it tolerates up to ~10% character error in the primary mission output. This is not a trivial formatting relaxation: for text-reading tasks (license plates, street names), a single character error can be operationally material, yet the success criterion explicitly forgives it. The paper defends this with 150 human-annotated comparisons (judge precision 1.0, recall 0.94 vs exact match recall 0.44), but 150 samples across heterogeneous reporting missions may not stabilize per-mission-type error rates, especially for long alphanumeric strings where 'one or two' differences are proportionally large. The load-bearing issue is not that the judge is unvalidated — it is that the paper's single headline number conflates 'semantic equivalence' with 'mission success,' and the choice materially changes the gap to human performance (human keyboard SR is 84.4% under the same judge? unclear whether humans use the same soft judge). If the success criterion is mission-specific reporting accuracy, the headline 'fewer than 35%' overstates capability under strict reading; if the criterion is semantic gist, the 34.8% is fair but the abstract's phrasing 'succeeds' is misleading. Either way, the central quantitative claim is not robust to the evaluation metric choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MissionBench introduces a benchmark and closed-loop evaluation protocol for frozen, off-the-shelf MLLMs acting as zero-shot aerial mission agents. The benchmark contains 120 missions across five Unreal Engine 5 / Cosys-AirSim environments and four task families (reporting, inspection, manipulation, patrol). Each agent receives a single natural-language instruction, an egocentric RGB image stream, pose/action history, and must output structured reasoning, bounding box, an action with continuous magnitude, an optional report, and a DONE flag. The paper evaluates 22 open- and closed-source MLLMs and reports that the strongest model, Gemini 3.1 Pro, succeeds on 34.8% of missions versus 84.4% for human keyboard control and 70.0% for humans using the same constrained VLM interface. Additional analyses compare success rate with oracle success rate, mission progress, collision rate, step efficiency, and a single-frame perception proxy, and ablate temperature, resolution, history length, and required output fields. The central claims are that mission-level aerial capability is far from human-level, that scaling improves zero-shot embodied performance, and that mission-level success requires capabilities beyond spatial perception.","tokens_in":23963,"tokens_out":6136,"duration_ms":67324,"significance":"If the measurement is accepted, MissionBench is a valuable contribution: it is, to my knowledge, the first aerial benchmark to combine continuous 4-DoF control, mission-level success criteria, and explicit reporting in a single closed-loop episode, and it evaluates a broad and current set of MLLMs. The paper's methodological strengths are considerable: human baselines under both natural keyboard control and the exact constrained MLLM interface; three runs per model with standard deviations reported in Appendix B.2; an LLM-as-judge validated against 150 human-annotated reporting cases; task-specific success criteria beyond simple distance thresholds; and a reserved held-out split for future fine-tuning. The failure-mode taxonomy and the weak correlation between a static perception proxy and mission-level performance support the paper's argument that closed-loop, mission-level evaluation reveals failures invisible to single-frame perception benchmarks. However, the headline quantitative claim is sensitive to the reporting success criterion, and the main results table contains an aggregation inconsistency that must be resolved before the numbers can be taken at face value.","major_comments":[{"comment":"The abstract's central claim that the strongest model 'succeeds on fewer than 35% of missions' is materially determined by the LLM-as-judge soft-matching rule for reporting missions. The paper itself states that replacing the judge with exact string matching drops Gemini 3.1 Pro's SR from 34.8% to 19.1%. The judge prompt in Table C6 explicitly permits 'if one or two numbers/letters differ' to count as success. For license-plate and street-name reporting, a one-character error is not a formatting artifact; it is a change in the task's correctness criterion. The 150-sample human validation (precision 1.00, recall 0.94 vs exact-match recall 0.44) is a genuine and commendable check, but it is not broken down by mission type or output-string length, and the human baselines in Table 3 are not reported under the exact-match criterion. Since reporting missions are 17/30 of the Test split, the he","section":"C.6 / Table C6 / Table 3 / Abstract"},{"comment":"The stated protocol is three runs per mission on a 30-mission test split, so each mission's success count is 0, 1, 2, or 3 out of 3, and the overall SR should be a multiple of 1/90 ≈ 1.11%. Several reported SR values cannot arise from this granularity: GPT-5.4 SR=2.0%±0.0, GPT-5.4 Mini SR=1.0%±2.2, and Gemini 3.1 Flash Lite SR=6.9%±0.0 are not integer multiples of 1/90. If the aggregation is per-mission (e.g., a mission counts as successful if any run succeeds, or if a majority succeeds), that convention must be stated explicitly; if the table reports averages of per-mission success rates, the granularity is different and the standard deviations need corresponding correction. This is not a cosmetic issue: the headline 34.8% and the per-model rankings are the core empirical payload of the paper.","section":"Table 3 / §4.1"}],"minor_comments":[{"comment":"Typos: 'angel' should be 'angle' in the line 'max_angle_step = 3 x {max_step_size}', and 'direclty' should be 'directly' in the format-reminder block. There is also 'in termns of' later in the per-step prompt.","section":"D.3, system prompt"},{"comment":"The perception proxy is measured on a 10-mission Proxy split, all in the Neighborhood environment and with mixed mission types, while the mission-level metrics used for correlation come from the 30-mission Test split with different environments and composition. Reporting confidence intervals or per-environment correlations would make the conclusion 'perception is not the primary bottleneck' more robust.","section":"§4.3 / Table B1"},{"comment":"Appendix A.1 appropriately acknowledges the limited scale and sim-to-real gap, but the main text and abstract do not qualify the 'fewer than 35%' claim with the soft-match caveat. A one-sentence qualification in the abstract or conclusion would align the headline with the evidence.","section":"A.1 / Conclusion"},{"comment":"Patrol and manipulation rows rest on only 3 missions each. For example, the human keyboard patrol SR of 88.9% corresponds to 8 of 9 episodes across 3 missions. The small per-type sample size should be noted in the main text when discussing task-family differences.","section":"Table B5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a serious benchmark contribution with careful human baselines and a thoughtful evaluation framework. The main blocking issue is the reporting success criterion: the headline number depends on an LLM-judge tolerance that is arguably too lenient for the very tasks the benchmark is designed to test. The second issue is a numerical inconsistency in the main table that needs clarification. If the authors add exact-match robustness reporting and fix the aggregation explanation, I would be supportive of acceptance. The paper seems well within the scope of a vision/robotics or AI journal; the 'uncontrolled capability emergence' discussion is appropriate and not overstated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading this one. First, the benchmark is a real contribution: it puts frozen MLLMs through whole missions with continuous 4-DoF control, mission-specific success criteria, explicit reporting, and closed-loop evaluation, which no prior aerial benchmark combines. Second, the headline claim—'strongest model succeeds on fewer than 35% of missions'—is true only under the LLM-as-judge soft-matching rule. Under exact string matching, Gemini 3.1 Pro's success rate drops from 34.8% to 19.1%. Reporting missions are 17 of the 30 test missions, so that choice carries the headline.\n\nWhat the paper does well: the benchmark design is thoughtful. Four task families (reporting, inspection, manipulation, patrol), task-specific success criteria, and metrics like OSR, MP, CR, and Eff let you separate navigation failure from last-mile perception and reporting failure. They run three seeds per model and report standard deviations in the appendix. The LLM-as-judge is validated against 150 human-annotated cases (precision 1.0, recall 0.94) and the paper honestly states that exact matching artificially suppresses success because of formatting equivalences. Human baselines under both the same VLM interface (70%) and continuous keyboard control (84.4%) give a fair comparison point. The perception proxy analysis—single-frame mIoU and depth RMSE only weakly predict mission success—is the most useful part; it supports the claim that mission-level competence needs more than spatial perception. The ablations on temperature, resolution, history length, and structured output are instructive.\n\nWhere the work is softest: the reporting success criterion. The judge prompt says 'if one or two numbers/letters differ, it can still be considered success.' For license plates and street names, a single character error can be operationally material. The judge is likely measuring semantic equivalence well, but the abstract's 'succeeds' conflates semantic gist with mission completion. This is not a fatal flaw—the human-model gap remains large under any scoring, and the paper's main qualitative conclusions hold—but the headline should be reported alongside the strict-match number or the judge's tolerance should be narrowed. Smaller issues: the test split has only three manipulation and three patrol missions, making those per-task numbers noisy; no code or environment assets are provided, which limits reproducibility; and the mission distribution skews heavily toward reporting.\n\nBottom line: this is a serious, honest benchmark paper. The central claim—zero-shot mission-level aerial control is hard, scaling helps, and single-frame perception is not the bottleneck—is supported by the data. The soft-matching issue is addressable and the paper already acknowledges part of it in the appendix. I'd send it to review.","headline":"MissionBench is a genuinely new closed-loop aerial benchmark with a solid empirical core, but the headline 34.8% success rate is softer than it looks because reporting missions are judged by an LLM that tolerates minor character errors.","tokens_in":24509,"tokens_out":2418,"would_cite":true,"duration_ms":27408,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces MissionBench and claims that state-of-the-art multimodal LLMs, evaluated zero-shot as closed-loop drone agents, complete fewer than 35% of missions compared to 84.4% human performance, and that mission-level competence","keywords":["MissionBench","aerial MLLM agents","zero-shot evaluation","closed-loop benchmark","embodied navigation","multimodal LLM","drone missions","scaling"],"falsifier":"Re-run MissionBench's 30-mission test split with a newer or larger MLLM; if any model exceeds 84.4% success, the central gap claim collapses. More locally, recompute success rates using exact string matching for reporting missions: the paper already reports this drops the best model from 34.8% to 19.1%, so the headline 'fewer than 35%' depends on the soft judge.","tokens_in":23430,"feed_emoji":"🚁","tokens_out":5659,"duration_ms":54968,"temperature":0.7,"pith_summary":"MissionBench is a closed-loop benchmark that asks general-purpose multimodal language models to fly a simulated drone from a single high-level instruction: they must decide where to go, how to position the camera, and what to report. Across 22 models, the strongest completes fewer than 35% of missions, while humans using the same interface complete 70% and humans with keyboard control complete 84.4%. The authors argue that mission-level competence is not explained by single-frame spatial perception alone; it requires integrating perception, planning, and reporting across a long-horizon episode. They also find that larger models perform better, indicating that general-purpose scaling contributes to embodied ability but remains far from human-level.","feed_headline":"Best AI drone agent succeeds on under 35% of missions","feed_subtitle":"New closed-loop benchmark shows scaling lifts MLLM agents but leaves them far below human mission success.","key_machinery":"The central object is MissionBench itself: 120 missions across five simulated 3D environments and four task families (reporting, inspection, manipulation, patrol), plus a closed-loop protocol in which the agent receives an egocentric image, emits a structured response (bounding box, reasoning, action with magnitude, report, and a DONE flag), the simulator executes the action, and the loop repeats until DONE or budget exhaustion. It introduces mission-level success criteria—spatial thresholds for inspection and manipulation, soft LLM-judged matching for reporting, IoU over buffered trajectories for patrol—and diagnostic metrics (Success Rate, Oracle Success Rate, Mission Progress, Collision R","core_discovery":"MissionBench shows that frozen, off-the-shelf MLLMs, evaluated zero-shot as aerial agents, succeed on at most 34.8% of 30 mission-level tasks, compared to 84.4% human performance with continuous control and 70.0% when humans are forced through the same text/image interface. The oracle success rate—whether the agent ever reaches the target—exceeds the success rate for several models, meaning agents often navigate to the right place but then fail to complete the mission through misidentification, premature termination, or drift. Scaling within model families consistently improves mission progress but does not close the human-model gap, and single-frame perception proxy scores correlate only mo","pith_inferences":["A direct extension: re-running the 30-mission test split with an explicit instruction to 'if you are close but cannot read the target, move closer before reporting' may lift SR for models with OSR > SR, isolating whether the bottleneck is policy or perception.","The LLM-as-judge's 90% similarity threshold is the most sensitive design choice; a public leaderboard reporting both soft and exact-match success rates would make the headline claim robust.","Because patrol missions reward 20m-buffer IoU against a human trajectory, a coverage-based metric (area actually swept) might change model rankings on route-following.","MissionBench missions start with the target partially visible, so results may understate the difficulty of pure search; extending to missions with no initial cue would test search ability separately."],"forward_implications":["If the 35% ceiling is real, general-purpose MLLMs are not yet reliable as zero-shot drone operators in simulated missions.","The consistent scaling trend suggests larger models will keep improving mission-level ability without task-specific training, which the paper flags as both an opportunity and a safety risk.","The OSR-SR gap implies that improving final reporting and termination decisions—knowing when to stop and what to report—may matter as much as navigation.","Closed-loop evaluation surfaces failure modes invisible to static perception benchmarks, so future embodied AI evaluations should adopt this protocol.","Existing mission-level benchmarks that decompose tasks into isolated subgoals cannot reveal these compounding failures; whole-mission evaluation is necessary."],"fun_headline_variants":["Top zero-shot AI drone agent fails 65% of missions","Aerial MLLMs stall at 35% mission success, humans hit 84%","MissionBench: zero-shot MLLMs score under 35% on aerial tasks","Scaling lifts MLLM agents, but mission success stays below 35%","AI drone agents: 35% mission success vs 84% human, scaling not enough"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Reporting missions are graded by an LLM judge that treats reports differing by one or two characters or numbers as successes (90% similarity threshold); if exact string matching is used instead, the best model's success rate drops from 34.8% to 19.1% (Appendix C.6).","fun_headline_variants_meta":{"raw":{"variants":["Top zero-shot AI drone agent fails 65% of missions","Aerial MLLMs stall at 35% mission success, humans hit 84%","MissionBench: zero-shot MLLMs score under 35% on aerial tasks","Scaling lifts MLLM agents, but mission success stays below 35%","AI drone agents: 35% mission success vs 84% human, scaling not enough"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1710,"prompt_tokens":718,"completion_tokens":992,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":886}},"tokens_in":462,"tokens_out":992,"duration_ms":7805,"temperature":1.0,"reasoning_tokens":886,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:02:46.421445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run MissionBench's 30-mission test split with a newer or larger MLLM; if any model exceeds 84.4% success, the central gap claim collapses. More locally, recompute success rates using exact string matching for reporting missions: the paper already reports this drops the best model from 34.8% to 19.1%, so the headline 'fewer than 35%' depends on the soft judge.","supporting_citations":[],"review_version":1}