Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

Current multimodal AI models are substantially better at reading the space around a drone than at understanding the drone's own motion and actions, and adding optical-flow cues narrows that gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:28 UTC pith:ACP5YTNQ

load-bearing objection A genuinely useful UAV-MLLM benchmark with a plausible but not yet airtight space/self gap; worth refereeing, needs uncertainty quantification and contamination analysis. the 4 major comments →

arxiv 2607.12477 v2 pith:ACP5YTNQ submitted 2026-07-14 cs.CV

Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence

Zhishan Zou , Guoyan Sun , Zhiwei Wei , Jiancheng Pan , Yujie Li , Mugen Peng , Wenjia Xu This is my paper
classification cs.CV
keywords self-in-spaceself-awarenessspatial cognitionUAV embodied intelligenceMLLM benchmarkingoptical flow fusionperception-memory-reasoningego-motion understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces SIS-Bench, a benchmark that asks whether vision-language AI models can jointly model the world around a drone and the drone itself. Across 13 tasks built from real drone videos, the benchmark measures two things: spatial cognition (objects, landmarks, layout) and self-awareness (the drone's own actions, motion history, and future behavior). The central finding is an imbalance: current models score consistently higher on space than on self, and scores fall steadily from perception to memory to reasoning. The paper also shows, in a controlled experiment, that feeding the model optical-flow motion cues raises both spatial and self scores, and that the improvement carries over to a downstream navigation task. If these results hold, the benchmark gives the field a new instrument for measuring the space–self gap and a simple cue that partly closes it.

Core claim

On the paper's own terms, the discovery is that contemporary multimodal language models are substantially stronger at modeling the external environment than at modeling the embodied agent itself: across all 26 evaluated models, spatial cognition scores exceed self-awareness scores, and both decline as tasks move from perception to memory to reasoning. The paper further claims that this imbalance is not fixed by simply scaling up models, because proprietary models' gains over open-source models are small on motion- and action-centric tasks. It then presents SIS-Motion, a controlled variant that fuses optical-flow-derived motion features with visual features, and reports that this raises the s

What carries the argument

The load-bearing object is the 'self-in-space' formulation: a benchmark organized along two axes — spatial cognition and self-awareness — and a three-level cognitive hierarchy of perception, memory, and reasoning. The second mechanism is the motion-aware fusion in SIS-Motion: optical-flow pseudo-images (magnitude plus x/y components) are encoded by a separate motion encoder and added to the visual tokens before the language model, giving the model an explicit signal about how the viewpoint itself is changing. This design converts the abstract space–self distinction into measurable QA tasks, and converts the remedy into a testable architectural change.

Load-bearing premise

The benchmark's self-awareness ground truth comes from reused action annotations in the source drone data and from language-model-generated metadata, not from independent flight telemetry, so systematically wrong or overly coarse action labels would make the measured space–self gap an artifact of labeling rather than a real property of the models.

What would settle it

Build a version of the self-awareness tasks whose answers are taken directly from flight-log telemetry (accelerometer, gyroscope, GPS or flight-controller state) and have two independent annotator pairs score the same questions; if models match their spatial-cognition scores on these telemetry-grounded questions, the paper's central imbalance claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • SIS-Bench provides a reusable instrument for measuring the gap between a model's understanding of a scene and its understanding of itself as an embodied agent, complementing environment-only UAV benchmarks.
  • The space-over-self imbalance appears across all 26 models and is not resolved by scale, so future UAV models need explicit self-state modeling rather than stronger recognition alone.
  • Optical-flow-based motion fusion yields consistent gains on perception- and memory-level tasks, suggesting that self-related dynamics are a usable supervisory signal for embodied video understanding.
  • The motion-aware gains transfer to a downstream UAV navigation decision task (92.2 percent vs. 71.2 percent for the backbone), so the effect is not confined to the benchmark.
  • The perception-to-memory-to-reasoning gradient means temporal and inferential integration is the hard part; simple motion cues only partially address it, leaving reasoning tasks like path planning below random in the controlled setup.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if self-awareness answers were rebuilt from true flight telemetry (IMU/GPS or flight-controller logs) instead of source action annotations and model-generated descriptions, the measured space–self gap could narrow or widen; the paper's conclusion about model limitations would be better grounded once label independence is verified.
  • Editorial inference: the asymmetry echoes a broad distinction between viewing a scene from outside and from the agent's own perspective; a testable extension is to add paused-frame ego-motion questions (pure rotation vs. translation) to isolate which motion types the model misreads.
  • Editorial inference: because optical flow helps perception and memory but not planning, a natural next step is fusing flow with explicit state estimators — velocity, heading, altitude — and testing long-horizon tasks; the paper's own limitations section points in this direction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This paper introduces SIS-Bench, a benchmark for evaluating UAV embodied intelligence along two dimensions (spatial cognition and self-awareness) and three cognitive levels (perception, memory, reasoning). The benchmark comprises 4,856 multiple-choice QA pairs derived from 1,646 real-world UAV videos through a task-conditioned pipeline with dual-expert verification. The authors evaluate 26 MLLMs (6 proprietary, 20 open-source) and report two main findings: models perform substantially better on spatial tasks than on self-awareness tasks, and performance degrades from perception to memory to reasoning. The paper then presents SIS-Motion, a motion-aware extension that fuses optical-flow features with visual tokens, and reports that it improves spatial and self averages from 72.0 to 74.2 and from 60.3 to 63.7 respectively over a visual-only SFT baseline, with transfer to a downstream OpenUAV navigation task.

Significance. If the findings hold, SIS-Bench would be a useful instrument for quantifying the space/self gap in embodied UAV intelligence, a dimension that existing UAV benchmarks largely ignore. The paper's strengths include the explicit dual-axis design, the three-level cognitive hierarchy, the use of heterogeneous real-world video sources, dual-expert verification with a documented modification/rejection rate, and broad model coverage (26 models) with additional robustness controls in Appendix D.1. The motion-aware study is a controlled exploration that isolates the effect of optical-flow injection via a visual-only SFT baseline. The downstream transfer experiment, however, currently lacks an SFT-only control, and the statistical treatment is thin. The central claim about a space/self imbalance rests on the validity and difficulty-matching of the self-awareness tasks, which is not yet established.

major comments (4)
  1. [App. B.2, Pipeline-A] The self-awareness ground truth for Action Recognition, Action Sequence, and Action Recall is 'directly reuse[d]' from AirScape action annotations, with GLM-4.6V-Flash-9B only normalizing labels. The paper reports no inter-annotator agreement for the subsequent expert review and no validation against IMU/GPS telemetry. Since the headline claim (Sec. 4.2) is that MLLMs are 'substantially stronger' at space than self, label noise or systematic ambiguity in the inherited action annotations could by itself produce or exaggerate the observed gap. Please report agreement statistics (e.g., Cohen's kappa) for the review process, validate a sample against telemetry, or otherwise demonstrate that the self-task labels are uniquely recoverable from videos.
  2. [Sec. 4.2, Table 1] The space/self comparison confounds construct with task difficulty. Human performance on Action Recognition (92.4%) is already below Object Attribute (96.6%), showing that self-tasks are intrinsically harder or more ambiguous. The claimed imbalance (e.g., Qwen2.5-VL-3B: 92.7 on Object Existence vs 37.9 on Action Recognition) may therefore reflect label/task asymmetry rather than a genuine self-awareness deficit. The paper needs either difficulty-matched task pairs, item-level IRT-style difficulty adjustment, or an explicit analysis that separates 'self-specific' difficulty from general task difficulty before the central claim can be accepted.
  3. [Tables 1 and 2] All accuracies are reported as single point estimates without error bars, confidence intervals, or significance tests. For the key motion-aware comparison, the 2.2 pp improvement in Spatial Avg and 3.4 pp in Self Avg (Table 2) could be within sampling noise; with roughly 4,856 items, the standard error is about 0.7 pp, but the per-task subsets are small (e.g., Relative Direction has 200 QA pairs) and multiple tasks are compared. Please provide bootstrap CIs or per-seed variance, and explicitly state which differences are statistically significant.
  4. [Sec. 5.4 / App. F] The downstream navigation transfer claim compares SIS-Motion (92.2%) against the vanilla Qwen2.5-VL 3B backbone (71.2%), but does not report the visual-only SFT baseline on the downstream task. Since SIS-Motion differs from the backbone by both fine-tuning and motion injection, the improvement cannot be attributed to motion-aware modeling without an SFT-only control. Please add the downstream accuracy of the visual-only SFT model under the same protocol.
minor comments (3)
  1. [Sec. C] The pixel bounds '256× 28 × 28 to 512 × 28 × 28' are confusing; likely a typo in rendering. Please clarify the intended input resolution.
  2. [Table 2 / Sec. 5.3] The claim that SIS-Motion 'improves Path Planning' from 20.6% to 23.5% is technically true but both values are below the 25.2% random baseline. D.3 explains this, but the main text should reference that caveat to avoid misleading readers.
  3. [Abstract / Sec. 3.3] The abstract says '1,646 real-world UAV videos' while Figure 3 and text also use 1,646; App. A.2 derives 689+467+237+253=1,646, which is consistent. Consider adding the video-type breakdown to the main text for clarity.

Circularity Check

0 steps flagged

No significant circularity: the benchmark findings are empirical evaluations, and the annotation pipeline is expert-corrected rather than self-defined.

full rationale

The paper's central claims are empirical benchmark results and a controlled fine-tuning comparison, not mathematical derivations. The load-bearing finding—that MLLMs are stronger at spatial cognition than self-awareness—rests on the SIS-Bench task suite. Although Pipeline-B uses GLM-4.6V-Flash-9B to draft spatial metadata, the paper states that 'These automatically produced metadata are then manually filtered and corrected before later QA construction,' and the overall review process modified 47.2% and rejected 21.3% of QA pairs. The final ground truth is therefore not equivalent to the annotating model's outputs by construction. Similarly, QA generation uses several LLMs, but the resulting questions pass dual-expert verification, so the test set is not simply the generating models' own output. The SIS-Motion exploration is controlled: both the visual-only SFT baseline and SIS-Motion are trained on the same SIS-Motion-54K corpus, isolating optical-flow fusion as the only difference, and training data are strictly separated from SIS-Bench. The paper's self-references in related work are not load-bearing. The limitations section explicitly acknowledges that the benchmark reorganizes established tasks rather than introducing new atomic tasks, which is a transparency statement rather than a circular step. No step in the paper reduces to its own input by definition or by a fitted-parameter renaming.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The central benchmark is an empirical evaluation artifact rather than a mathematical derivation, so the main burden is on label reliability and evaluation validity. The only hand-fitted numerical parameters are the downstream navigation thresholds; the axioms listed above are domain assumptions about ground truth, proxy validity, and input modality.

free parameters (1)
  • Downstream navigation label thresholds = altitude change >5 m -> takeoff/landing; turn angle 25-120 deg -> turn; otherwise cruise
    These thresholds define the ground-truth action segments for the OpenUAV-derived downstream test (Sec. F). They are chosen by hand without sensitivity analysis, so the reported transfer accuracy depends on them.
axioms (5)
  • domain assumption AirScape action annotations accurately describe UAV self-motion and are valid ground truth for self-awareness tasks.
    Pipeline-A 'directly reuses the available action annotations' (App. B.2). If the source action labels are noisy, use a different taxonomy, or do not correspond to actual ego-motion, all self-awareness conclusions are affected.
  • domain assumption VLM-generated scene metadata, after manual correction, is reliable ground truth for spatial cognition tasks.
    Pipeline-B uses GLM-4.6V-Flash-9B to produce spatial metadata (App. B.2); the same model family is later evaluated on tasks derived from that metadata. Manual filtering and dual-expert review mitigate but do not eliminate the risk of systematic label bias.
  • domain assumption Multiple-choice QA performance is a valid proxy for embodied spatial intelligence and self-awareness.
    SIS-Bench evaluates all 13 tasks as 4-option multiple-choice questions with open-loop video inputs; App. G.1 acknowledges this limitation, so the benchmark measures a proxy rather than closed-loop embodied behavior.
  • domain assumption Visual-only observation with at most 32 frames is an appropriate test bed for UAV self-awareness.
    Evaluation protocols in App. C cap inputs at 32 frames and give no IMU, GPS, or gimbal telemetry. The conclusion that MLLMs struggle with self-awareness is therefore conditional on this visual-only setting.
  • domain assumption Optical-flow pseudo-images encode agent dynamics well enough for language-model fusion.
    The SIS-Motion design (Sec. 5.1) assumes that converting optical flow into 3-channel pseudo-images and passing them through a ViT motion encoder provides useful self-related cues; the ablation across flow estimators supports this, but it remains a representation assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 32654 in / 13201 out tokens · 130652 ms · 2026-08-02T06:28:20.911857+00:00 · methodology

0 comments
read the original abstract

Autonomous UAV systems increasingly rely on multimodal large language models (MLLMs) to operate in complex real-world environments. Such embodied scenarios require not only understanding the surrounding space but also maintaining a coherent representation of the agent itself. However, existing UAV-oriented approaches and benchmarks remain largely environment-centric, primarily focusing on spatial understanding tasks, with the agent's self-awareness remaining implicit. To address this gap, we introduce SIS-Bench, a benchmark for evaluating embodied spatial intelligence in UAV scenarios under a unified self-in-space formulation. SIS-Bench organizes evaluation along two complementary dimensions, space and self, and a three-level hierarchy of perception, memory, and reasoning. It contains 4,856 question--answer pairs across 13 tasks derived from 1,646 real-world UAV videos through a task-conditioned construction pipeline with expert verification. Extensive evaluations reveal that current MLLMs exhibit fundamental limitations in modeling dynamic and agent-centered processes. In particular, we observe a clear imbalance between spatial cognition and self-awareness, as well as a progressive performance degradation across cognitive levels. Motivated by these findings, we further explore a motion-aware representation that incorporates self-related dynamics through optical flow and visual feature fusion. Experimental results show that modeling agent motion consistently improves perception and memory performance, not only in spatial cognition but also in self-awareness, and generalizes to downstream UAV decision-making tasks. Our results highlight the importance of self-awareness for advancing embodied spatial intelligence, and provide both a new benchmark and empirical evidence for motion-aware self-in-space modeling.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs

    cs.CV 2026-07 conditional novelty 6.0

    UAV-DualCog evaluates multimodal LLMs on self-state and environment-state reasoning in aerial images and videos, and shows current models are unreliable at spatial and temporal grounding.

Reference graph

Works this paper leans on

74 extracted references · 19 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Uav-based studies in railway infrastructure monitoring.Automation in Construction, 167:105714, 2024

    Peyman Aela, Hung-Lin Chi, Ali Fares, Tarek Zayed, and Minkoo Kim. Uav-based studies in railway infrastructure monitoring.Automation in Construction, 167:105714, 2024

    2024

  2. [2]

    Qwen3.5-plus, 2026

    Alibaba Cloud. Qwen3.5-plus, 2026. URLhttps://www.alibabacloud.com/help/en/model-studio/models

    2026

  3. [3]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3674–3683, 2018

    2018

  4. [4]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Doubao-seed-1.6-vision, 2025

    ByteDance Seed. Doubao-seed-1.6-vision, 2025. URLhttps://research.doubao.com/zh/seed1_6

    2025

  7. [7]

    Seed-1.8, 2025

    ByteDance Seed. Seed-1.8, 2025. URLhttps://github.com/ByteDance-Seed/Seed-1.8

    2025

  8. [8]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024

    2024

  9. [9]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26428–26438, 2024

    2024

  10. [10]

    Grounded 3d-llm with referent tokens.arXiv preprint arXiv:2405.10370, 2024

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens.arXiv preprint arXiv:2405.10370, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Mm-uavbench: How well do multimodal large language models see, think, and plan in low-altitude uav scenarios?arXiv preprint arXiv:2512.23219, 2025

    Shiqi Dai, Zizhi Ma, Zhicong Luo, Xuesong Yang, Yibin Huang, Wanyue Zhang, Chi Chen, Zonghao Guo, Wang Xu, Yufei Sun, et al. Mm-uavbench: How well do multimodal large language models see, think, and plan in low-altitude uav scenarios?arXiv preprint arXiv:2512.23219, 2025

    arXiv 2025

  13. [13]

    Memflow: Optical flow estimation and prediction with memory

    Qiaole Dong and Yanwei Fu. Memflow: Optical flow estimation and prediction with memory. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19068–19078, 2024

    2024

  14. [14]

    Large language model-assisted uav operations and communications: A multifaceted survey and tutorial

    Yousef Emami, Hao Zhou, Radha Reddy, Atefeh Hajijamali Arani, Biliang Wang, Kai Li, Luis Almeida, and Zhu Han. Large language model-assisted uav operations and communications: A multifaceted survey and tutorial. arXiv preprint arXiv:2602.19534, 2026

    arXiv 2026

  15. [15]

    Speaker-follower models for vision-and-language navigation.Advances in neural information processing systems, 31, 2018

    Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and-language navigation.Advances in neural information processing systems, 31, 2018

    2018

  16. [16]

    Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Objectrelator: Enabling cross-view object relation understanding across ego-centric and exo-centric perspectives

    Yuqian Fu, Runze Wang, Bin Ren, Guolei Sun, Biao Gong, Yanwei Fu, Danda Pani Paudel, Xuanjing Huang, and Luc Van Gool. Objectrelator: Enabling cross-view object relation understanding across ego-centric and exo-centric perspectives. InICCV, 2025

    2025

  18. [18]

    Openfly: A comprehensive platform for aerial vision-language navigation

    Yunpeng Gao, Chenhui Li, Zhongrui You, Junli Liu, Zhen Li, Pengan Chen, Qizhi Chen, Zhonghan Tang, Liansheng Wang, Penghui Yang, et al. Openfly: A comprehensive platform for aerial vision-language navigation. arXiv preprint arXiv:2502.18041, 2025

    arXiv 2025

  19. [19]

    Gemini 3 flash, 2026

    Google. Gemini 3 flash, 2026. URLhttps://ai.google.dev/gemini-api/docs/models. 13

    2026

  20. [20]

    Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.arXiv preprint arXiv:2507.01006, 2025

    Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.arXiv preprint arXiv:2507.01006, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36: 20482–20494, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36: 20482–20494, 2023

    2023

  22. [22]

    Step3-vl-10b technical report.arXiv preprint arXiv:2601.09668, 2026

    Ailin Huang, Chengyuan Yao, Chunrui Han, Fanqi Wan, Hangyu Guo, Haoran Lv, Hongyu Zhou, Jia Wang, Jian Zhou, Jianjian Sun, et al. Step3-vl-10b technical report.arXiv preprint arXiv:2601.09668, 2026

    arXiv 2026

  23. [23]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems, 37:113991–114017, 2024

    2024

  24. [24]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Hierarchical cross-modal agent for robotics vision- and-language navigation

    Muhammad Zubair Irshad, Chih-Yao Ma, and Zsolt Kira. Hierarchical cross-modal agent for robotics vision- and-language navigation. In2021 IEEE international conference on robotics and automation (ICRA), pages 13238–13246. IEEE, 2021

    2021

  26. [26]

    Leveraging drones for effective disaster management: A comprehensive analysis of the 2024 noto peninsula earthquake case in japan.Progress in Disaster Science, 23:100348, 2024

    Mikio Ishiwatari. Leveraging drones for effective disaster management: A comprehensive analysis of the 2024 noto peninsula earthquake case in japan.Progress in Disaster Science, 23:100348, 2024

    2024

  27. [27]

    Embodiment in multimodal large language models.arXiv preprint arXiv:2510.13845, 2025

    Akila Kadambi, Lisa Aziz-Zadeh, Antonio Damasio, Marco Iacoboni, and Srini Narayanan. Embodiment in multimodal large language models.arXiv preprint arXiv:2510.13845, 2025

    arXiv 2025

  28. [28]

    Agentic uavs: Llm-driven autonomy with integrated tool-calling and cognitive reasoning.arXiv preprint arXiv:2509.13352, 2025

    Anis Koubaa and Khaled Gabr. Agentic uavs: Llm-driven autonomy with integrated tool-calling and cognitive reasoning.arXiv preprint arXiv:2509.13352, 2025

    arXiv 2025

  29. [29]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  30. [30]

    Embodied agent interface: Benchmarking llms for embodied decision making

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li E Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking llms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428–100534, 2024

    2024

  31. [31]

    Spatial intelligence in vision-language models: A comprehensive survey.TechRxiv, 2025

    Disheng Liu, Tuo Liang, Zhe Hu, Jierui Peng, Yiren Lu, Yi Xu, Yun Fu, and Yu Yin. Spatial intelligence in vision-language models: A comprehensive survey.TechRxiv, 2025. doi: 10.36227/techrxiv.176231405.57942913/v2

    arXiv 2025

  32. [32]

    Remoteclip: A vision language foundation model for remote sensing.IEEE Transactions on Geoscience and Remote Sensing, 62:1–16, 2024

    Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Remoteclip: A vision language foundation model for remote sensing.IEEE Transactions on Geoscience and Remote Sensing, 62:1–16, 2024

    2024

  33. [33]

    Indooruav: Benchmarking vision-language uav navigation in continuous indoor environments

    Xu Liu, Yu Liu, Hanshuo Qiu, Yang Qirong, and Zhouhui Lian. Indooruav: Benchmarking vision-language uav navigation in continuous indoor environments. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 23864–23872, 2026

    2026

  34. [34]

    Ovis2.5 technical report.arXiv preprint arXiv:2508.11737, 2025

    Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jianshan Zhao, et al. Ovis2.5 technical report.arXiv preprint arXiv:2508.11737, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Exo2egosyn: Unlocking foundation video generation models for exocentric-to-egocentric video synthesis.arXiv preprint arXiv:2511.20186, 2025

    Mohammad Mahdi, Yuqian Fu, Nedko Savov, Jiancheng Pan, Danda Pani Paudel, and Luc Van Gool. Exo2egosyn: Unlocking foundation video generation models for exocentric-to-egocentric video synthesis.arXiv preprint arXiv:2511.20186, 2025

    arXiv 2025

  36. [36]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16488–16498, 2024

    2024

  37. [37]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshkulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 14

    2023

  38. [38]

    Scale and multiple psychologies of space

    Daniel R Montello. Scale and multiple psychologies of space. InEuropean conference on spatial information theory, pages 312–321. Springer, 1993

    1993

  39. [39]

    Gpt-5.4, 2026

    OpenAI. Gpt-5.4, 2026. URLhttps://openai.com/index/introducing-gpt-5-4/

    2026

  40. [40]

    V2-sam: Marrying sam2 with multi-prompt experts for cross-view object correspondence

    Jiancheng Pan, Runze Wang, Tianwen Qian, Mohammad Mahdi, Yanwei Fu, Xiangyang Xue, Xiaomeng Huang, Luc Van Gool, Danda Pani Paudel, and Yuqian Fu. V2-sam: Marrying sam2 with multi-prompt experts for cross-view object correspondence. InCVPR, 2026

    2026

  41. [41]

    Gpt4scene: Understand 3d scenes from videos with vision-language models.arXiv preprint arXiv:2501.01428, 2025

    Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models.arXiv preprint arXiv:2501.01428, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  43. [43]

    Uav-vln: End-to-end vision language guided navigation for uavs

    Pranav Saxena, Nishant Raghuvanshi, and Neena Goveas. Uav-vln: End-to-end vision language guided navigation for uavs. In2025 European Conference on Mobile Robots (ECMR), pages 1–6. IEEE, 2025

    2025

  44. [44]

    Videoflow: Exploiting temporal cues for multi-frame optical flow estimation

    Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 12469–12480, 2023

    2023

  45. [45]

    Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report.arXiv preprint arXiv:2504.07491, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Cai, Yuan Cao, Y

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, S.H. Cai, Yuan Cao, Y. Charles, H.S. Che, Cheng Chen, Guanduo Chen, et al. Kimi k2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    Pith/arXiv arXiv 2026

  47. [47]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. InEuropean conference on computer vision, pages 402–419. Springer, 2020

    2020

  48. [48]

    Uavscenes: A multi-modal dataset for uavs

    Sijie Wang, Siqi Li, Yawei Zhang, Shangshu Yu, Shenghai Yuan, Rui She, Quanjiang Guo, JinXuan Zheng, Ong Kang Howe, Leonrich Chandra, et al. Uavscenes: A multi-modal dataset for uavs. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 28946–28958, 2025

    2025

  49. [49]

    Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Towards realistic uav vision-language navigation: Platform, benchmark, and methodology.arXiv preprint arXiv:2410.07087, 2024

    Xiangyu Wang, Donglin Yang, Ziqin Wang, Hohin Kwan, Jinyu Chen, Wenjun Wu, Hongsheng Li, Yue Liao, and Si Liu. Towards realistic uav vision-language navigation: Platform, benchmark, and methodology.arXiv preprint arXiv:2410.07087, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Sea-raft: Simple, efficient, accurate raft for optical flow

    Yihan Wang, Lahav Lipson, and Jia Deng. Sea-raft: Simple, efficient, accurate raft for optical flow. InEuropean Conference on Computer Vision, pages 36–54. Springer, 2024

    2024

  52. [52]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769, 2023

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes.arXiv preprint arXiv:2308.08769, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Skyscript: A large and semantically diverse vision-language dataset for remote sensing

    Zhecheng Wang, Rajanie Prabha, Tianyuan Huang, Jiajun Wu, and Ram Rajagopal. Skyscript: A large and semantically diverse vision-language dataset for remote sensing. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5805–5813, 2024

    2024

  54. [54]

    Visdrone-mot2019: The vision meets drone multiple object tracking challenge results

    Longyin Wen, Pengfei Zhu, Dawei Du, Xiao Bian, Haibin Ling, Qinghua Hu, Jiayu Zheng, Tao Peng, Xinyao Wang, Yue Zhang, et al. Visdrone-mot2019: The vision meets drone multiple object tracking challenge results. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019

    2019

  55. [55]

    Aerial vision-language navigation with a unified framework for spatial, temporal and embodied reasoning.arXiv preprint arXiv:2512.08639, 2025

    Huilin Xu, Zhuoyang Liu, Yixiang Luomei, and Feng Xu. Aerial vision-language navigation with a unified framework for spatial, temporal and embodied reasoning.arXiv preprint arXiv:2512.08639, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025

    2025

  57. [57]

    Mimo-vl technical report.arXiv preprint arXiv:2506.03569, 2025

    Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, et al. Mimo-vl technical report.arXiv preprint arXiv:2506.03569, 2025. 15

    Pith/arXiv arXiv 2025

  58. [58]

    Why do mllms struggle with spatial understanding? a systematic analysis from data to architecture.arXiv preprint arXiv:2509.02359, 2025

    Wanyue Zhang, Yibin Huang, Yangbin Xu, JingJing Huang, Helu Zhi, Shuo Ren, Wang Xu, and Jiajun Zhang. Why do mllms struggle with spatial understanding? a systematic analysis from data to architecture.arXiv preprint arXiv:2509.02359, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Urbanvideo-bench: Benchmarking vision-language models on embodied intelligence with video data in urban spaces

    Baining Zhao, Jianjie Fang, Zichao Dai, Ziyou Wang, Jirong Zha, Weichen Zhang, Chen Gao, Yue Wang, Jinqiang Cui, Xinlei Chen, et al. Urbanvideo-bench: Benchmarking vision-language models on embodied intelligence with video data in urban spaces. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2025

  60. [60]

    Airscape: An aerial generative world model with motion controllability

    Baining Zhao, Rongze Tang, Mingyuan Jia, Ziyou Wang, Fanhang Man, Xin Zhang, Yu Shang, Weichen Zhang, Wei Wu, Chen Gao, et al. Airscape: An aerial generative world model with motion controllability. InProceedings of the 33rd ACM International Conference on Multimedia, pages 12519–12528, 2025

    2025

  61. [61]

    Video-3d llm: Learning position-aware video representation for 3d scene understanding

    Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video representation for 3d scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8995–9006, 2025

    2025

  62. [62]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Pith/arXiv arXiv 2024

  63. [63]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 16 Appendix AirScape UrbanVideo-Bench VisDrone Frame 0 Frame 32 Frame 64 Frame 96 Frame 128 Frame 0...

    Pith/arXiv arXiv 2025

  64. [64]

    Strictly prohibit choosing cars, pedestrians, or animals

    Object Selection: Must choose permanently fixed static landmarks (buildings, sculptures, streetlights, trees). Strictly prohibit choosing cars, pedestrians, or animals

  65. [65]

    Visual features (color, shape, material) must be used for identification

    De-orientation Description: In the descriptions of `anchor_object` and `target_object`, it is strictly prohibited to use any directional words (e.g., the building on the left, the tower in the distance). Visual features (color, shape, material) must be used for identification

  66. [66]

    Annotation Steps (Step-by-Step)

    Temporal Stability: Within the selected time window, the relative positional relationship between the two must remain unchanged. Annotation Steps (Step-by-Step)

  67. [67]

    Select Anchor: Find the most prominent and easily described static object (e.g., a red pavilion)

  68. [68]

    Select Target: Find another static object (e.g., a white stone bridge)

  69. [69]

    Determine Relation: Judge which direction the Target is relative to the Anchor (e.g., the white bridge is Behind the red pavilion)

  70. [70]

    scene_summary

    Find Distractors: List 3-4 objects that truly exist in the scene but are NOT in that orientation (e.g., a tree on the left, a streetlight on the right). You are a highly rigorous expert in UAV spatial relation analysis. The current task is: Identify and annotate the relative spatial orientation between static landmarks in the input video. All output conte...

  71. [71]

    on the left

    Cleaning Check: Options must NOT contain words like "on the left" or "in the background." They must be pure noun phrases (e.g., " Autumn-colored trees")

  72. [72]

    Figure 12QA-generation template, illustrated with theRelative Directiontask

    Random Distribution: The correct answer must be randomly assigned to A, B, C, or D. Figure 12QA-generation template, illustrated with theRelative Directiontask. We design one template for each of the 13 tasks and use multiple LLMs to improve the diversity of question phrasing and distractor construction while preserving answer uniqueness. B.3 QA generatio...

  73. [971]

    moving forward

    Among the 13 tasks,Action Recall,Action Recognition,Object Existence, andLandmark Recallare the largest subsets, while the reasoning tasks remain comparatively compact because they require heavier expert annotation. B.6 Benchmark comparison Table 6 compares SIS-Bench with representative UAV/aerial benchmarks. The first three benchmarks in the table, namel...

    2048

  74. [2340]

    At the cognitive-level breakdown, perception contains 1765 QA pairs, memory 2120, and reasoning