{"id":"1450bfdd-abd2-4f10-b491-7c39cabe5976","arxiv_id":"2505.08765","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper introduces CityAVOS, the first city-space benchmark for UAV visual object search, and PRPSearcher, an MLLM-based agent that outperforms baselines but not humans.","lead":"A new simulated benchmark, CityAVOS, tests drones that must find everyday objects in city streets using only an image and a text description. The authors' MLLM-powered agent, PRPSearcher, beats baseline search methods by a large margin but still trails human pilots.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Success is self-declared: CityAVOS counts an episode as successful when the agent itself stops and claims identification, with no independent ground-truth check described, so the headline 53.50% SR may include hallucinated or premature successes.","rationale":"The reader's weakest assumption identifies the same core issue I consider most load-bearing: the success metric depends on the agent's own determination of success, with no independent verification described. That is a direct threat to the central claim that CityAVOS provides a standardized evaluation and that PRPSearcher achieves 53.50% SR. I also noticed a separate, objectively checkable problem in Eq. 19: the SPL formula as printed is not the standard SPL and appears unbounded, which would undermine the reported SPL numbers if taken literally. However, I treat the SPL formula as a secondary issue because it may be a typographical error in the appendix and because SR is the more central metric. The paper's contribution is plausible and the experiments are internally consistent, but the evaluation protocol needs clarification and a verification step before the headline numbers can be trusted. Since this is addressable by re-running with an oracle checker, the conditional verdict stands rather than a full rejection. The proposed concrete test directly settles whether the self-report concern is real: if an oracle checker changes the numbers substantially, the central claim is inflated; if it does not, the concern is resolved and the results can be accepted with the metric definition fixed.","tokens_in":19620,"tokens_out":5337,"duration_ms":59126,"concrete_test":"Add an oracle success checker to the simulator: after any stop action, compute the ground-truth 3D position of the target object and require both (a) Euclidean distance <= 20 m and (b) the target instance is actually visible in the final RGB-D observation, e.g., by projecting its ground-truth bounding box and requiring a minimum mask/box overlap with the agent's segmentation output. Set the success flag from this checker, not from the agent's self-report. Rerun PRPSearcher, STMR, and the baseline methods on the same 605 episodes and recompute SR, SPL (using the standard per-episode SPL formula), MSS, and NE. If corrected SR drops materially or the PRPSearcher-versus-STMR gap narrows, the headline claim needs revision. In addition, rerun the theta_T sweep on a held-out split of episodes rather than on the same episodes used for the final numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that 'the agent determines whether to search and locate the target successfully based on observations,' and Appendix A.2 says the MLLM-based agent decides whether the target is found and then executes the stop action. Appendix A.3.1 defines the success flag fs_i as a Boolean but does not specify who or what sets it. Because no oracle, post-hoc verifier, or ground-truth overlap check is described, every counted success rests on the agent's own stop/identification decision. In an urban benchmark full of visually similar shops, signs, and buildings, an MLLM that confidently stops at a look-alike object, or announces the target while within 20 m of the wrong instance, would still be scored as a success. This directly threatens the headline 53.50% SR and the +37.69% SR margin over baselines. Separately, Eq. 19 writes SPL as SR * sum(tl_i / tl*_i), which is unbounded and is not the standard SPL definition; taken literally, the reported 40.57 and all SPL comparisons are not well-defined. Both issues need to be pinned down before the central performance claims can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CityAVOS, a synthetic urban benchmark for aerial visual object search (AVOS) built on the EmbodiedCity simulator, containing 2,420 tasks over six object categories and three difficulty levels. It also proposes PRPSearcher, an MLLM-based agent that constructs an object-centric 3D semantic map, a cognitive map driven by MLLM-assigned attraction values, and a 3D uncertainty map, and that uses an IPT prompting mechanism to balance exploration and exploitation. The authors report that PRPSearcher achieves 53.50% SR and 40.57% SPL on CityAVOS, outperforming several adapted indoor and outdoor baselines, and include ablations over the semantic-map design, the exploration/exploitation components, the IPT threshold, and the choice of MLLM.","tokens_in":19911,"tokens_out":3487,"duration_ms":39261,"significance":"If the reported results hold, the paper would provide a useful standardized benchmark for a relatively underexplored task and a competitive MLLM-agent baseline that is openly described. The construction of a 2,420-task dataset with manual verification, the public code/dataset link, and the ablations are genuine strengths. However, the central performance claims are currently not established: the success criterion is not independently verified, the SPL formula as written is not the standard metric and is not well defined, and the headline numbers are single point estimates with the exploration threshold selected on the same subset used for evaluation. These issues are load-bearing because they directly affect the claimed 53.50% SR and the reported margins over baselines; they are, however, fixable within the scope of the manuscript.","major_comments":[{"comment":"The success flag fs_i is never tied to ground-truth object positions. Section 3.1 states that 'the agent determines whether to search and locate the target successfully based on observations,' Appendix A.2 says the MLLM-based agent decides whether the target has been found and then executes the stop action, and Appendix A.3.1 defines fs_i only as a Boolean without specifying who or what sets it. Appendix A.3.5 mentions 'subsequent verification' in one case study, but no verification protocol is described. Because every counted success therefore rests on the agent's own stop/identification decision, the headline 53.50% SR and the +37.69% margin over baselines may include hallucinated or premature stops. Please add an offline success checker that compares the stop position and detected object against the recorded ground-truth P_object, run it for all methods, and report the corrected metrics.","section":"Section 3.1, Appendix A.2, Appendix A.3.1"},{"comment":"The SPL formula SPL = SR * sum_i (tl_i / tl*_i) is not the standard success-weighted path-length metric and is not well defined as written: it multiplies a fraction by a sum of per-episode ratios, so the result can exceed 1 and is not a mean over episodes, and it applies the path ratio to failed episodes as well. This directly affects the reported SPL values of 40.57 for PRPSearcher and all SPL comparisons in Table 2. Please replace it with the standard per-episode definition, typically SPL = (1/q) * sum_i fs_i * (tl*_i / max(tl_i, tl*_i)), clip or otherwise handle tl*_i = 0, and re-report all SPL numbers.","section":"Appendix A.3.1, Eq. (19)"},{"comment":"All experimental numbers are single point estimates with no variance, repeated runs, or confidence intervals, and the IPT threshold theta_T is tuned on the same 605-task subset that is used for the final evaluation. Table 4 shows a clear performance peak at theta_T = 0.1, but because the same data select and then report this value, the magnitude of the reported improvements and the claim that this setting is 'optimal' are not statistically supported. Please add multiple runs or seeds with standard errors, use a validation split for threshold selection, and state how many episodes each aggregated cell in Tables 2-5 is based on.","section":"Section 5.2, Section 5.3, Table 4"}],"minor_comments":[{"comment":"The total MSS for PRPSearcher is reported as 35.26 in Table 2 and 35.20 in Table 3; please make the numbers consistent.","section":"Table 2 vs Table 3"},{"comment":"The NE formula uses fp*_i but the surrounding text defines the ground-truth position as tp*; please unify the notation and clarify that tl*_i is the shortest-path length to the target, not the recorded trajectory length.","section":"Appendix A.3.1, Eq. (18)"},{"comment":"The threshold is written as theta in Eq. (15) but as theta_T in Table 4 and the surrounding text; please use one symbol throughout and define its units or range.","section":"Section 4.4 and Table 4"},{"comment":"The dataset collection is described as manually verified, but no inter-annotator agreement or quality-control statistics are reported; a sentence describing how many trajectories were rejected or regenerated would strengthen confidence in the benchmark labels.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible benchmark-and-agent contribution, but the experimental protocol needs substantial strengthening before publication. The success-verification issue and the SPL definition are the two points I would ask the authors to address first; if an oracle-based success check and a corrected SPL metric change the reported numbers materially, the paper's central claim will need to be rewritten accordingly. The dataset release and code link are a positive feature, but the anonymous repository should be checked to confirm it contains the task files and the evaluation harness described in the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this paper gives the UAV embodied-AI community a decent first benchmark but its headline performance claims are not yet trustworthy because success is self-declared by the agent and the SPL formula is misdefined.\n\nWhat is actually new: CityAVOS is the first city-scale, instruction-free aerial visual object search benchmark I know of, and that alone fills a real gap. The task design — six object categories, three difficulty levels, unique vs non-unique targets — is sensible, and the adaptation of EmbodiedCity into 2,420 tasks is a solid piece of work. PRPSearcher is a reasonable MLLM-based agent: the three-map architecture (semantic, cognitive, uncertainty) is a genuine new combination, and the ablations support the design choices. The paper is also honest about baseline adaptations, including noting that Say-REAPEx and NEUSIS were not open-sourced. That transparency earns credit.\n\nThe soft spots are concentrated in the evaluation. Section 3.1 says the agent itself determines whether it has located the target; Appendix A.2 confirms the MLLM decides and executes the stop action; Appendix A.3.1 defines the success flag but does not say who sets it. No oracle, no post-hoc verification, no ground-truth overlap check. In a city full of visually similar shops and signs, an agent that stops near the wrong instance or hallucinates a match would still be counted as successful. That directly threatens the 53.50% SR headline and the large margin over baselines. This is a load-bearing issue, though not a fatal one — it can be fixed with an independent verification step.\n\nEquation 19 is a separate, concrete error: SPL = SR * sum(tl_i / tl*_i) is not the standard SPL, it is unbounded and not averaged over episodes. The reported SPL numbers are therefore not well-defined. Minor fix, but it must be corrected.\n\nAlso worth noting: theta_T is tuned on the same benchmark used for final evaluation, and only 25% of tasks (605) are used, with no analysis of sampling. No reruns or variance estimates. All addressable.\n\nOverall: this is a useful contribution for the UAV embodied-AI subfield, and it deserves serious peer review. The benchmark is valuable, the method is plausible, and the internal comparisons are consistent. I would not desk-reject it. I would ask the authors to (1) add an independent success-verification mechanism, (2) correct the SPL definition, and (3) either move tuning to a held-out split or justify tuning on the test set. Once those are pinned down, the performance claims can be taken seriously.","headline":"A genuinely useful first benchmark for instruction-free aerial visual object search, but the headline SR/SPL numbers rest on self-declared success and a misdefined SPL until the evaluation protocol is fixed.","tokens_in":20422,"tokens_out":2568,"would_cite":false,"duration_ms":23929,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that CityAVOS is the first benchmark for autonomous UAV visual object search in urban spaces, and that its PRPSearcher agent reaches 53.50% success rate, far above baselines.","keywords":["Aerial Visual Object Search","UAV navigation","multi-modal large language model","benchmark dataset","semantic mapping","exploration-exploitation balance","urban embodied intelligence","spatial reasoning"],"falsifier":"Re-run PRPSearcher on the same 605-task subset and have an independent verifier compare the object at the stop location with the ground-truth target; if the verified success rate is substantially below the reported 53.50%, the headline result is inflated by self-reported stops.","tokens_in":19428,"feed_emoji":"🚁","tokens_out":5953,"duration_ms":49316,"temperature":0.7,"pith_summary":"This paper introduces CityAVOS, the first benchmark for Aerial Visual Object Search (AVOS) in city spaces, containing 2,420 tasks over six object categories and three difficulty levels. To solve these tasks, it proposes PRPSearcher, an agent built on multimodal large language models that constructs three maps: object-centric semantic, attraction-based cognitive, and uncertainty maps, plus an IPT prompting mechanism to balance exploration and exploitation. On a 605-task subset, PRPSearcher achieves 53.50% success rate and 40.57% SPL, substantially exceeding the compared baselines. The reported gap to human performance (78.68% SR) indicates that semantic reasoning and spatial exploration in AVOS remain open problems.","feed_headline":"New benchmark and agent push UAV city search to 53.5% success","feed_subtitle":"CityAVOS brings 2,420 urban search tasks; the MLLM agent PRPSearcher triples baseline success but trails humans.","key_machinery":"The load-bearing mechanism is the triple-map architecture carried through the agent's loop. The object-centric 3D dynamic semantic map restricts semantic segmentation to objects the MLLM judges relevant to the target, keeping the map sparse. The 3D cognitive map stores an attraction value $A(s)\\in[0,1]$ per semantic category, so high-attraction clusters (found by DBSCAN) mark likely target locations; a mirrored map zeroes out already-recognized grids, which is the denoising step. The 3D uncertainty map attenuates each cell's uncertainty by $f(d)=e^{-\\alpha d}$ with distance, so the exploration advice is the action maximizing total uncertainty reduction. The Inspiration Promote Thought (IPT) prompt injects that exploration advice only when the reward exceeds threshold $\\theta$, letting the MLLM planner combine long-term exploitation guidance with occasional 'inspiration' to explore.","core_discovery":"The central claim is that autonomous UAV search for common urban objects, given only an image and text description, can be formalized as a benchmark task and solved better by an MLLM-driven agent than by existing navigation methods. CityAVOS provides the standardized tasks, and PRPSearcher demonstrates the approach: it selectively segments only target-related semantics into a 3D dynamic semantic map, assigns MLLM-derived attraction values to build a 3D cognitive map, tracks exploration with a 3D uncertainty map, and uses thresholded exploration advice in its prompts. The paper reports that this system reaches 53.50% SR and 40.57% SPL on CityAVOS, outperforming indoor-adapted and outdoor baselines by clear margins, while still trailing human operators.","pith_inferences":["The paper's success metric relies on the agent's own stop decision without independent verification, so a re-evaluation with a post-hoc object check would likely yield a lower verified success rate; this should be treated as a caveat when comparing headline numbers.","Because dataset labels were manually verified without reported inter-annotator agreement, some hard tasks may contain ambiguous or noisy labels; measuring agreement would strengthen the benchmark's reliability.","The agent depends on commercial MLLM APIs at every reasoning step, which is costly and slow; distilling the map-building and prompting policy into a smaller model is a natural testable extension.","The synthetic Unreal Engine environment may not capture real-world visual noise and drone dynamics, so a sim-to-real transfer study on real drone footage is an open question the paper does not address."],"forward_implications":["If these results hold, CityAVOS becomes a standard testbed for comparing UAV object-search agents, filling the gap left by indoor-only object navigation and instruction-based aerial navigation benchmarks.","PRPSearcher's object-centric semantic segmentation and attraction-based cognitive map suggest that restricting perception to target-related semantics is key to efficient search in cluttered urban scenes.","The ablation results imply that neither pure exploration nor pure exploitation is sufficient; the IPT threshold $\\theta$ provides a tunable balance that peaks at $\\theta=0.1$.","The performance gap between the best agent and human operators (53.50% vs 78.68% SR) points to concrete next targets: better semantic reasoning about sparse cues and more efficient spatial exploration."],"supporting_citations":[{"why":"Provides the high-fidelity urban simulator platform the CityAVOS dataset is built on.","marker":"[13]"},{"why":"Adds realistic UAV flight and sensing simulation to the CityAVOS environment.","marker":"[26]"},{"why":"Supplies the Grounded-SAM semantic segmentation model used to build the semantic map.","marker":"[2]"},{"why":"Clusters high-attraction grids in the cognitive map to identify likely target regions.","marker":"[25]"},{"why":"One of the indoor-adapted baselines that PRPSearcher outperforms on CityAVOS.","marker":"[40]"},{"why":"A curiosity-driven baseline adapted for the benchmark comparison.","marker":"[21]"},{"why":"The strongest baseline among compared methods, representing outdoor aerial vision-language navigation with top-down semantic mapping.","marker":"[15]"}],"fun_headline_variants":["CityAVOS: 2,420 tasks benchmark UAV search; PRPSearcher tops baselines","PRPSearcher MLLM agent boosts UAV object search on new CityAVOS benchmark","UAV search benchmark CityAVOS: agent beats baselines but humans still win","CityAVOS: 2,420 tasks; PRPSearcher leads but humans ahead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline success metric counts an episode as successful when the agent itself stops within 20 meters and says it has found the target, with no independent check that the object is the actual target.","fun_headline_variants_meta":{"raw":{"variants":["CityAVOS: 2,420 tasks benchmark UAV search; PRPSearcher tops baselines","PRPSearcher MLLM agent boosts UAV object search on new CityAVOS benchmark","UAV search benchmark CityAVOS: agent beats baselines but humans still win","CityAVOS: 2,420 tasks; PRPSearcher leads but humans ahead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001383,"raw_usage":{"total_tokens":5655,"prompt_tokens":1052,"completion_tokens":4603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":4511}},"tokens_in":668,"tokens_out":4603,"duration_ms":28537,"temperature":1.0,"reasoning_tokens":4511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:47:08.862346+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run PRPSearcher on the same 605-task subset and have an independent verifier compare the object at the stop location with the ground-truth target; if the verified success rate is substantially below the reported 53.50%, the headline result is inflated by self-reported stops.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Adds realistic UAV flight and sensing simulation to the CityAVOS environment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Grounded-SAM semantic segmentation model used to build the semantic map."}],"review_version":1}