The reviewed record of science sign in
Pith

arxiv: 2607.06537 · v1 · pith:N4VP26JF · submitted 2026-07-07 · cs.RO

UniLM-Nav: A Unified Framework for Zero-Shot Last-Mile Navigation

Zhuofan Zhang , Tianxu Wang , Guoxi Zhang , Yixiong Lin , Xilin Wang , Hongming Xu , Qing Li , Song-Chun Zhu
show 1 more author
Lifeng Fan
This is my paper

Reviewed by Pith2026-07-08 02:24 UTCglm-5.2pith:N4VP26JFopen to challenge →

classification cs.RO
keywords last-mile navigationmobile manipulationmultimodal large language modelzero-shot navigationaffordance groundingbase-pose reasoningopen-vocabulary manipulationOVMM benchmark
0
0 comments X

The pith

Zero-Shot Robot Navigation Closes the Last Mile with MLLM Spatial Reasoning

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes UniLM-Nav, a zero-shot framework that uses a shared multimodal large language model (MLLM) to solve last-mile navigation — the problem of moving a robot from the vicinity of a target object to a manipulation-ready base pose. The framework decomposes last-mile navigation into three sequential stages: view selection from a short-term memory buffer, task-conditioned affordance grounding to identify a pixel-level interaction point, and geometry-aware base-pose reasoning that lifts the 2D affordance into 3D robot-centric coordinates and predicts a feasible base position. All three stages are resolved by prompting the same MLLM backend without any task-specific training or manual pose annotation. On the OVMM benchmark, UniLM-Nav achieves 23.77% overall success rate, outperforming the prior best method by 3.13 percentage points. The authors also deploy the system on a Unitree B2 quadruped with a Z1 arm, achieving 52.5% total success across four real-world mobile manipulation tasks. A key design finding is that explicitly decomposing the problem into simpler MLLM calls outperforms asking the model to jointly perform view selection and affordance grounding, and that providing explicit 3D geometric context outperforms asking the MLLM to directly infer base positions from 2D images.

Core claim

The central mechanism is the decomposition of last-mile navigation into three sequential, explicitly separated MLLM reasoning stages — view selection, affordance grounding, and geometry-aware base-pose reasoning — each of which constrains the next. The paper demonstrates that this decomposition is not merely an engineering convenience but is load-bearing: merging view selection and affordance grounding into a single MLLM call degrades performance across all tested backends, and replacing geometry-aware base-pose reasoning with direct 2D floor-point grounding substantially reduces success. Additionally, the paper finds that geometrically computing the robot heading (orienting toward the 3D-lu

What carries the argument

Three-stage MLLM decomposition: (1) view selection from K-step memory buffer, (2) task-conditioned affordance grounding producing normalized 2D pixel coordinates, (3) geometry-aware base-pose reasoning using depth-lifted 3D affordance point and robot configuration to predict (x, y) position with geometrically computed heading θ.

If this is right

  • The decomposition principle — splitting embodied reasoning into sequential, single-purpose MLLM calls rather than monolithic joint inference — may generalize to other embodied AI tasks where current MLLMs struggle with multi-constraint reasoning in a single step.
  • The finding that a 4B robotics-fine-tuned MLLM outperforms much larger general-purpose models suggests that domain-specific spatial training data may matter more than raw parameter count for embodied reasoning tasks.
  • The explicit 3D lifting step (converting 2D affordance points to robot-centric coordinates via depth) sidesteps MLLMs' known weakness in metric distance estimation, offering a design pattern for other embodied systems that need geometric precision without training custom models.
  • The system's 61% failure rate from upstream navigation errors suggests that last-mile navigation improvements are necessary but not sufficient for mobile manipulation, and that future work on active local exploration before last-mile reasoning could substantially improve end-to-end success.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The ablation showing that thinking models help most at the base-pose reasoning stage but can hurt at the affordance grounding stage suggests that different embodied subtasks may have distinct optimal reasoning profiles — some benefitting from extended chain-of-thought, others from direct perception. This implies that future embodied MLLM systems might route different subtasks to different model co
  • The real-world success rate dropping sharply on spatial-relation tasks (e.g., 'in front of the monitor', 'bottom-left corner') versus simple pick-and-place suggests that MLLM spatial reasoning remains the binding constraint for fine-grained manipulation instructions, and that benchmark success rates on simpler tasks may overestimate real-world deployability.

Load-bearing premise

The framework assumes that upstream object navigation reliably brings the robot to within 1–2 meters of the target with the target visible in recent observations. The authors' own error analysis shows that 61% of failures originate from navigation errors occurring before last-mile navigation even begins, meaning the system's success is heavily gated by a component the framework does not improve.

What would settle it

If a single-stage MLLM call (jointly performing view selection, affordance grounding, and base-pose reasoning) matched or exceeded the three-stage decomposition's success rate across multiple backends and environments, the central design claim — that explicit decomposition is necessary because current MLLMs cannot reliably handle these tasks jointly — would be undermined.

Figures

Figures reproduced from arXiv: 2607.06537 by Guoxi Zhang, Hongming Xu, Lifeng Fan, Qing Li, Song-Chun Zhu, Tianxu Wang, Xilin Wang, Yixiong Lin, Zhuofan Zhang.

Figure 1
Figure 1. Figure 1: The robot is tasked with placing the bottle on the table in front of the monitor. Object-goal [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: UniLM-Nav decomposes last-mile navigation into view selection, affordance ground [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Framework ablation of UniLM-Nav. (a) Success rates of UniLM-Nav and its variants on [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Left: Manual error breakdown on the 20% OVMM subset used in our ablation studies. Right: Representative last-mile navigation failures involving view selection, affordance grounding, and base-pose reasoning. The green annotations indicate preferred views, affordance regions, or base poses, while the red annotations show the corresponding erroneous predictions [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: View Selection Prompt for Pick and Place [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Affordance Grounding Prompt for Pick and Place [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Base-pose Reasoning Prompt for Pick and Place [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative example of view selection from the short-term memory [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative success case on the OVMM benchmark. After object navigation ends, UniLM-Nav predicts the affordance point in the first-person view, shown as the red point, and infers the robot’s base position conditioned on the grounded affordance, task instruction, and robot geometry, shown as the green point. By executing last-mile navigation based on the predicted base position, the robot successfully adjus… view at source ↗
Figure 11
Figure 11. Figure 11: Real-world experiment: Pick up the cup from the table. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Real-world experiment: Place the cake on the plate. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Real-world experiment: Place the bottle on the table, in front of the monitor. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Real-world experiment: Imagine facing the monitor, place the cake at the bottom-left [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
read the original abstract

Mobile manipulation requires a robot to navigate to a target object or receptacle and then perform intended manipulation. However, reaching the vicinity of the target does not guarantee a manipulation-ready base pose, a problem known as last-mile navigation. Prior methods for last-mile navigation either rely on manual pose annotation or task-specific training, limiting their scalability to open-vocabulary settings with fine-grained spatial constraints. We propose UniLM-Nav, a unified framework for zero-shot open-vocabulary last-mile navigation. UniLM-Nav decomposes last-mile navigation into view selection, task-conditioned affordance grounding, and geometry-aware base-pose reasoning, all resolved with a shared multimodal large language model (MLLM) backend. Specifically, UniLM-Nav first selects a reference view that best captures the target object or receptacle from recently collected observations. It then grounds task-relevant affordance point in the selected view and lifts the result into the robot-centric coordinate frame. Finally, conditioned on the grounded affordance, task context, and robot geometry, it infers a manipulation-ready base pose for the robot. We evaluate UniLM-Nav on the OVMM benchmark, where it outperforms the previous state-of-the-art method, MoTo, by 3.13 percentage points. Analyses show that the components of our method are crucial to final performance, and that the choice of MLLM also has a substantial effect. We further deploy UniLM-Nav on a Unitree B2 quadruped robot with a 6-DoF Unitree Z1 manipulator, validating its applicability to real-world mobile manipulation tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. The paper proposes UniLM-Nav, a zero-shot MLLM-based framework for last-mile navigation in mobile manipulation. The framework decomposes the problem into three stages—view selection from a short-term memory buffer, task-conditioned affordance grounding, and geometry-aware base-pose reasoning—all handled by a shared MLLM backend. The method is evaluated on the OVMM benchmark, reporting 23.77% Overall SR (a 3.13pp improvement over MoTo), and is further validated on a real quadruped manipulator across four tasks. Ablation studies on a 20% subset examine the contribution of each component, the choice of MLLM backend, and thinking vs. non-thinking variants. The approach is well-motivated and the decomposition into explicit geometric reasoning steps is a sensible design choice.

Significance. The paper addresses a genuine gap in mobile manipulation—the navigation-to-manipulation handoff—and proposes a clean, training-free formulation using MLLMs. The explicit decomposition into view selection, affordance grounding, and geometry-aware base-pose reasoning is a reasonable architectural choice, and the ablations (Tables 4–6) provide useful empirical insights, including the finding that embodied-tuned models (RoboBrain-2.5-4B) can outperform much larger general-purpose models. The real-world deployment on a quadruped manipulator adds practical value. However, the headline SOTA claim is not adequately supported by statistical evidence, and the real-world validation tests an augmented pipeline not used in the benchmark evaluation, which limits the significance of the transferability claim.

major comments (3)
  1. §5.1, Table 1: The headline claim that UniLM-Nav achieves state-of-the-art performance, outperforming MoTo by 3.13 percentage points (23.77% vs. 20.64%), is reported without any variance estimates, confidence intervals, or multiple-seed results. The OVMM validation set size is not stated in the paper, but if it is on the order of 200 episodes (as is typical for this benchmark), the standard error of the difference between two proportions of this magnitude is approximately 4.2pp, meaning the observed 3.13pp gap is within one standard error and cannot be distinguished from sampling noise. The authors should either (a) report the number of evaluation episodes and provide a statistical test or confidence interval, or (b) temper the SOTA claim to acknowledge that the improvement is not established as statistically significant. This is load-bearing because the SOTA claim is the paper's central
  2. §5.2, Fig. 3a and associated text: The ablation studies are conducted on a 20% subset of the OVMM validation set, where the full UniLM-Nav method achieves 25.42% Overall SR (Table 2, Gemini-3-Flash-Preview row). On the full validation set (Table 1), the same configuration achieves 23.77%. This 1.65pp discrepancy is not discussed. While subset variance is expected, the authors should acknowledge this gap and note that the ablation magnitudes (e.g., the 5pp drop from removing view selection, 25.42% to 20.42%) are measured on the subset and may not exactly reflect full-set effects. This matters because the ablation conclusions are load-bearing for the claim that each component contributes.
  3. Appendix B.6: The real-world experiments include an additional affordance refinement step (re-querying the MLLM with the wrist-camera image after the robot reaches the predicted base pose) that is not part of the OVMM benchmark pipeline described in §4–§5. This means the hardware validation tests an enhanced system rather than the UniLM-Nav framework as evaluated on the benchmark. The authors should explicitly disclose this discrepancy in §5.4 (not only in the appendix) and clarify that the real-world results reflect the framework plus an additional refinement step, so that readers do not interpret the hardware results as direct evidence of benchmark performance.
minor comments (7)
  1. §5.1: The number of OVMM validation episodes used for the main results is not stated anywhere in the paper. This should be reported in the experimental setup.
  2. Table 1 caption: 'Gemini-3-Flash-Previw' is a typo; should be 'Gemini-3-Flash-Preview'.
  3. §4, Eq. (2): The affordance grounding output is described as a normalized 2D coordinate in (0,1), but Appendix A.2 notes that some backends use a 0–1000 coordinate space. The main text should briefly mention this convention difference.
  4. §5.4, Table 3: The 'Total Success' row reports 52.5%, but the four task rates (70%, 60%, 40%, 40%) average to 52.5%. It would be clearer to label this as 'Average Success Rate' or show the raw counts (21/40) to confirm the total.
  5. Fig. 3a: The y-axis label and tick values are not legible in the provided PDF. The figure should be regenerated at higher resolution.
  6. §3: The assumption that object navigation reliably brings the robot to a near-target state (1–2 meters) is acknowledged as a limitation, but the error analysis (Appendix B.4) shows navigation errors account for 61% of failures. This dependency should be more prominently discussed in the main text, not only in the appendix, as it contextualizes the system's ceiling performance.
  7. References: Several model references (e.g., GPT-5.4, Gemini-3-Flash-Preview, Qwen3-VL) point to product pages or preprints that may not be stable at the time of publication. Where possible, permanent archival references should be used.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. All three major comments are well-taken. We will (1) report the OVMM validation set size and temper the SOTA claim, (2) acknowledge the subset–full-set discrepancy in the ablation discussion, and (3) disclose the real-world affordance refinement step in the main text. We agree with every point; the revisions are straightforward.

read point-by-point responses

  1. Referee: §5.1, Table 1: The headline claim that UniLM-Nav achieves state-of-the-art performance, outperforming MoTo by 3.13 percentage points (23.77% vs. 20.64%), is reported without any variance estimates, confidence intervals, or multiple-seed results. The OVMM validation set size is not stated in the paper, but if it is on the order of 200 episodes (as is typical for this benchmark), the standard error of the difference between two proportions of this magnitude is approximately 4.2pp, meaning the observed 3.13pp gap is within one standard error and cannot be distinguished from sampling noise. The authors should either (a) report the number of evaluation episodes and provide a statistical test or confidence interval, or (b) temper the SOTA claim to acknowledge that the improvement is not established as statistically significant. This is load-bearing because the SOTA claim is the paper's central

    Authors: The referee is correct on both the factual gap (we did not report the number of evaluation episodes) and the statistical point (a 3.13pp difference on ~200 episodes is within one standard error). We will address this in two ways. First, we will state the OVMM validation set size explicitly in §5.1. Second, we will temper the SOTA claim throughout the paper. In the abstract, we will change the claim from 'outperforms the previous state-of-the-art method, MoTo, by 3.13 percentage points' to 'achieves 23.77% Overall SR, improving over the previous state-of-the-art method, MoTo (20.64%), though the difference is not established as statistically significant given the benchmark size.' In §5.1, we will add a sentence noting that the improvement is within sampling noise for a benchmark of this size and should be interpreted as a competitive result rather than a statistically significant advance. We will retain the absolute numbers for transparency but will not claim statistical significance. revision: yes

  2. Referee: §5.2, Fig. 3a and associated text: The ablation studies are conducted on a 20% subset of the OVMM validation set, where the full UniLM-Nav method achieves 25.42% Overall SR (Table 2, Gemini-3-Flash-Preview row). On the full validation set (Table 1), the same configuration achieves 23.77%. This 1.65pp discrepancy is not discussed. While subset variance is expected, the authors should acknowledge this gap and note that the ablation magnitudes (e.g., the 5pp drop from removing view selection, 25.42% to 20.42%) are measured on the subset and may not exactly reflect full-set effects. This matters because the ablation conclusions are load-bearing for the claim that each component contributes.

    Authors: The referee is right that this discrepancy should be acknowledged. The 1.65pp gap between the 20% subset (25.42%) and the full validation set (23.77%) is expected given subset variance, but we should have stated this explicitly. We will add a note in §5.2 after the ablation setup description, stating that the subset results may differ from full-set performance due to sampling variance, and that the ablation magnitudes should be interpreted as indicative of component contributions rather than as precise full-set effect sizes. We will also add a cross-reference to Table 1 so readers can directly compare the subset and full-set numbers. revision: yes

  3. Referee: Appendix B.6: The real-world experiments include an additional affordance refinement step (re-querying the MLLM with the wrist-camera image after the robot reaches the predicted base pose) that is not part of the OVMM benchmark pipeline described in §4–§5. This means the hardware validation tests an enhanced system rather than the UniLM-Nav framework as evaluated on the benchmark. The authors should explicitly disclose this discrepancy in §5.4 (not only in the appendix) and clarify that the real-world results reflect the framework plus an additional refinement step, so that readers do not interpret the hardware results as direct evidence of benchmark performance.

    Authors: The referee is correct. The real-world pipeline includes an affordance refinement step (re-querying the MLLM with the wrist-camera image after the robot reaches the predicted base pose) that is not part of the OVMM benchmark pipeline. This step is described in Appendix B.6 but is not mentioned in §5.4, which could mislead readers into thinking the hardware results directly reflect the benchmark-evaluated system. We will add a sentence in §5.4 disclosing that the real-world deployment includes an additional affordance refinement step using the wrist camera, motivated by real-world odometry and calibration errors, and that this step is not part of the OVMM benchmark pipeline. We will make clear that the hardware results should be interpreted as evidence of real-world applicability of the framework plus this refinement, not as a direct replication of the benchmark configuration. revision: yes

Circularity Check

0 steps flagged

No circularity found. The framework is zero-shot with no fitted parameters, evaluated on an external benchmark with independent baselines.

full rationale

The paper's derivation chain is straightforward and non-circular. UniLM-Nav decomposes last-mile navigation into three MLLM calls (Eqs. 1-3): view selection, affordance grounding, and base-pose reasoning. Each step takes externally-provided inputs (RGB-D observations, task instructions, robot configuration, depth-derived 3D coordinates) and produces outputs via zero-shot prompting of an off-the-shelf MLLM. No parameter is fitted to the evaluation data; the framework is explicitly zero-shot. The main results (Table 1) are evaluated on the OVMM benchmark [23] whose metrics are defined by the benchmark organizers, and baselines (HomeRobot, MoManipVLA, MoTo, UniTeam) are from independent groups. Ablations (Tables 4-6, Fig. 3) remove framework components and measure effects on the same externally-defined metric. No self-citation is load-bearing for the central claims: the authors cite their own prior work [30] (M2 Diffuser) only in the related-work section, not as a premise. The RoboBrain models [24, 54] used as MLLM backends are from a different group. The skeptic's concerns about statistical significance of the 3.13pp improvement and the extra affordance refinement step in real-world experiments (Appendix B.6) are validity and reproducibility concerns, not circularity — they do not involve any step where an output reduces to an input by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper introduces no new physical entities, particles, forces, or mathematical objects. The framework is composed of existing components (MLLMs, depth cameras, navigation stacks) combined in a pipeline. The free parameters are engineering heuristics (K=5, 70-80% arm reach, 0.2m gripper offset) rather than fitted scientific constants.

free parameters (3)
  • Short-term memory buffer size K = 5
    Set by default in §A.1. Not tuned or justified beyond 'default' choice.
  • Optimal manipulation distance heuristic = 70-80% of max arm reach
    Stated in the base-pose reasoning prompt (Fig. 7). This is a hand-specified heuristic, not derived from data.
  • Gripper height offset = 0.2 meters
    Used in the arm lift computation formula in the base-pose reasoning prompt (Fig. 7). A fixed constant added to the target height.
axioms (3)
  • domain assumption Object navigation reliably brings the robot to a near-target state where the target is visible in recent observations.
    Stated in §3 and §6 (Limitations). The entire pipeline depends on this input condition being met.
  • domain assumption MLLMs can reliably select better views and ground affordance points from single RGB images when prompted with structured instructions.
    The method's viability depends on MLLM visual reasoning being sufficient for these sub-tasks. The error analysis (§5.3) shows this is imperfect.
  • domain assumption The geometric computation of heading (orienting robot toward the lifted affordance point) is superior to MLLM-predicted heading.
    Validated empirically in Table 5, but adopted as a design axiom rather than derived from first principles.

pith-pipeline@v1.1.0-glm · 23757 in / 2587 out tokens · 367424 ms · 2026-07-08T02:24:43.325605+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 71 canonical work pages · 8 internal anchors

  1. [1]

    R. Yang, Y . Kim, R. Hendrix, A. Kembhavi, X. Wang, and K. Ehsani. Harmonic mobile manipulation. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3658–3665. IEEE, 2024

    work page 2024

  2. [2]

    Z. Wu, Y . Zhou, X. Xu, Z. Wang, and H. Yan. Momanipvla: Transferring vision-language- action models for general mobile manipulation. InThe IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1714–1723, 2025

    work page 2025

  3. [3]

    A. Xiao, N. Janaka, T. Hu, A. Gupta, K. Li, C. Yu, and D. Hsu. Robi butler: Multimodal remote interaction with a household robot assistant.arXiv preprint arXiv:2409.20548, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Abbatematteo, E

    B. Abbatematteo, E. Rosen, S. Thompson, T. Akbulut, S. Rammohan, and G. Konidaris. Com- posable interaction primitives: A structured policy class for efficiently learning sustained- contact manipulation skills. In2024 IEEE international conference on robotics and automation (ICRA), pages 7522–7529. IEEE, 2024

    work page 2024

  5. [5]

    Y . Peng, Z. Wang, Y . Zhang, S. Zhang, N. Cai, F. Wu, and M. Chen. Revolutionizing battery disassembly: The design and implementation of a battery disassembly autonomous mobile manipulator robot (beam-1). In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6367–6374. IEEE, 2024

    work page 2024

  6. [6]

    ˇStibinger, G

    P. ˇStibinger, G. Broughton, F. Majer, Z. Rozsyp´alek, A. Wang, K. Jindal, A. Zhou, D. Thakur, G. Loianno, T. Krajn´ık, et al. Mobile manipulator for autonomous localization, grasping and precise placement of construction material in a semi-structured environment.IEEE Robotics and Automation Letters (RA-L), 6(2):2595–2602, 2021

    work page 2021

  7. [7]

    Chang, T

    M. Chang, T. Gervet, M. Khanna, S. Yenamandra, D. Shah, S. Y . Min, K. Shah, C. Paxton, S. Gupta, D. Batra, et al. Goat: Go to any thing. InRobotics: Science and Systems (RSS),

    work page

  8. [8]

    doi:10.15607/RSS.2024.XX.073

    work page doi:10.15607/rss.2024.xx.073 2024

  9. [9]

    Zhang, X

    P. Zhang, X. Gao, Y . Wu, K. Liu, D. Wang, Z. Wang, B. Zhao, Y . Ding, and X. Li. Moma- kitchen: A 100k+ benchmark for affordance-grounded last-mile navigation in mobile manipu- lation. InInternational Conference on Computer Vision (ICCV), pages 6315–6326, 2025

    work page 2025

  10. [10]

    R.-Z. Qiu, Y . Song, X. Peng, S. A. Suryadevara, G. Yang, M. Liu, M. Ji, C. Jia, R. Yang, X. Zou, et al. Wildlma: Long horizon loco-manipulation in the wild. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 10011–10019. IEEE, 2025

    work page 2025

  11. [11]

    Huang, D

    X. Huang, D. Batra, A. Rai, and A. Szot. Skill transformer: A monolithic policy for mobile manipulation. InInternational Conference on Computer Vision (ICCV), pages 10852–10862, 2023

    work page 2023

  12. [12]

    Yokoyama, A

    N. Yokoyama, A. Clegg, J. Truong, E. Undersander, T.-Y . Yang, S. Arnaud, S. Ha, D. Batra, and A. Rai. Asc: Adaptive skill coordination for robotic mobile manipulation.IEEE Robotics and Automation Letters (RA-L), 9(1):779–786, 2023

    work page 2023

  13. [13]

    J. Yang, I. Huang, B. Vu, M. Bajracharya, R. Antonova, and J. Bohg. Mobi-π: Mobilizing your robot learning policy. InConference on Robot Learning (CoRL), volume 305 ofProceedings of Machine Learning Research, pages 3516–3536. PMLR, 2025. 9

    work page 2025

  14. [14]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything. InInternational Conference on Computer Vision (ICCV), pages 4015–4026, 2023

    work page 2023

  15. [15]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, et al. DINOv2: Learning robust visual features without supervi- sion.Transactions on Machine Learning Research, 2024

    work page 2024

  16. [16]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational Conference on Machine Learning (ICML), volume 139 ofProceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021

    work page 2021

  17. [17]

    GPT-4o System Card

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Openai o3 and o4-mini system card.https://openai.com/index/ o3-o4-mini-system-card/, 2025

    OpenAI. Openai o3 and o4-mini system card.https://openai.com/index/ o3-o4-mini-system-card/, 2025

    work page 2025

  19. [19]

    Z. Wu, A. Ma, X. Xu, H. Yin, Y . Liang, Z. Wang, J. Lu, and H. Yan. Moto: A zero-shot plug-in interaction-aware navigation for general mobile manipulation. InConference on Robot Learning (CoRL), volume 305 ofProceedings of Machine Learning Research, pages 2933–

    work page

  20. [20]

    Lin, J.-F

    T.-J. Lin, J.-F. Yeh, H.-T. Su, C.-Y . Lin, Y .-T. Chen, and W. H. Hsu. Affordance-guided coarse- to-fine exploration for base placement in open-vocabulary mobile manipulation. InAAAI Con- ference on Artificial Intelligence (AAAI), pages 18443–18451, 2026

    work page 2026

  21. [21]

    Z. Cai, Y . Wang, Q. Sun, R. Wang, C. Gu, W. Yin, Z. Lin, Z. Yang, C. Wei, O. Qian, et al. Holis- tic evaluation of multimodal llms on spatial intelligence.arXiv preprint arXiv:2508.13142, 2025

    work page arXiv 2025

  22. [22]

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    Gemini 3 flash: frontier intelligence built for speed.https://blog.google/ products-and-platforms/products/gemini/gemini-3-flash/, 2025

    Google. Gemini 3 flash: frontier intelligence built for speed.https://blog.google/ products-and-platforms/products/gemini/gemini-3-flash/, 2025

    work page 2025

  24. [24]

    Yenamandra, A

    S. Yenamandra, A. Ramachandran, K. Yadav, A. S. Wang, M. Khanna, T. Gervet, T.-Y . Yang, V . Jain, A. Clegg, J. M. Turner, et al. Homerobot: Open-vocabulary mobile manipulation. InConference on Robot Learning (CoRL), volume 229 ofProceedings of Machine Learning Research, pages 1975–2011. PMLR, 2023

    work page 1975

  25. [25]

    H. Tan, E. Zhou, Z. Li, Y . Xu, Y . Ji, X. Chen, C. Chi, P. Wang, H. Jia, Y . Ao, et al. Robobrain 2.5: Depth in sight, time in mind.arXiv preprint arXiv:2601.14352, 2026

    work page arXiv 2026

  26. [26]

    Introducing gpt-5.4.https://openai.com/index/introducing-gpt-5-4/, 2026

    OpenAI. Introducing gpt-5.4.https://openai.com/index/introducing-gpt-5-4/, 2026

    work page 2026

  27. [27]

    Unitree b2.https://unitree.com/b2, 2026

    Unitree Robotics. Unitree b2.https://unitree.com/b2, 2026. Product page, accessed 2026-06-02

    work page 2026

  28. [28]

    Unitree z1.https://unitree.com/z1, 2026

    Unitree Robotics. Unitree z1.https://unitree.com/z1, 2026. Product page, accessed 2026-06-02

    work page 2026

  29. [29]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems (RSS), 2023. doi:10.15607/RSS.2023.XIX.025. 10

    work page doi:10.15607/rss.2023.xix.025 2023

  30. [30]

    Z. Fu, T. Z. Zhao, and C. Finn. Mobile ALOHA: Learning bimanual mobile manipulation using low-cost whole-body teleoperation. InConference on Robot Learning (CoRL), volume 270 ofProceedings of Machine Learning Research, pages 4066–4083. PMLR, 2025

    work page 2025

  31. [31]

    S. Yan, Z. Zhang, M. Han, Z. Wang, Q. Xie, Z. Li, Z. Li, H. Liu, X. Wang, and S.-C. Zhu. M2 diffuser: Diffusion-based trajectory optimization for mobile manipulation in 3d scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–17, 2025. doi: 10.1109/TPAMI.2025.3553454

    work page doi:10.1109/tpami.2025.3553454 2025

  32. [32]

    P. Liu, Y . Orru, J. Vakil, C. Paxton, N. M. M. Shafiullah, and L. Pinto. Demonstrating OK- Robot: What really matters in integrating open-knowledge models for robotics. InRobotics: Science and Systems (RSS), 2024. doi:10.15607/RSS.2024.XX.091

    work page doi:10.15607/rss.2024.xx.091 2024

  33. [33]

    Rosen, S

    E. Rosen, S. James, S. Orozco, V . Gupta, M. Merlin, S. Tellex, and G. Konidaris. Synthesizing navigation abstractions for planning with portable manipulation skills. InConference on Robot Learning (CoRL), volume 229 ofProceedings of Machine Learning Research, pages 2278–

    work page

  34. [34]

    F. Wang, S. Lyu, P. Zhou, A. Duan, G. Guo, and D. Navarro-Alarcon. Instruction-augmented long-horizon planning: Embedding grounding mechanisms in embodied mobile manipulation. InAAAI Conference on Artificial Intelligence (AAAI), pages 14690–14698, 2025

    work page 2025

  35. [35]

    Quartey, E

    B. Quartey, E. Rosen, S. Tellex, and G. Konidaris. Verifiably following complex robot in- structions with foundation models. InInternational Conference on Robotics and Automation (ICRA), pages 1–8. IEEE, 2025

    work page 2025

  36. [36]

    T.-H. Lee, F. Mahmudova, and K. Desingh. Learning category-level last-meter navigation from rgb demonstrations of a single-instance.arXiv preprint arXiv:2512.11173, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    L. Qin, M. Wang, P. Li, W. Zhou, and H. Li. Active perception meets rule-guided rl: A two-phase approach for precise object navigation in complex environments. InInternational Conference on Computer Vision (ICCV), pages 7603–7612, 2025

    work page 2025

  38. [38]

    K. Chai, H. Lee, and J. J. Lim. N2m: Bridging navigation and manipulation by learning pose preference from rollout.arXiv preprint arXiv:2509.18671, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  39. [39]

    Nasiriany, F

    S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. InInternational Conference on Machine Learning (ICML), volume 235 ofProceedings of Machine Learning Research, pages 37321–37341. PMLR, 2024

    work page 2024

  40. [40]

    K. Fang, F. Liu, P. Abbeel, and S. Levine. MOKA: Open-world robotic manipulation through mark-based visual prompting. InRobotics: Science and Systems (RSS), 2024. doi:10.15607/ RSS.2024.XX.062

    work page 2024

  41. [41]

    Y . Tang, S. Zhang, X. Hao, P. Wang, J. Wu, Z. Wang, and S. Zhang. Affordgrasp: In-context affordance reasoning for open-vocabulary task-oriented grasping in clutter. InInternational Conference on Intelligent Robots and Systems (IROS), pages 9433–9439. IEEE, 2025

    work page 2025

  42. [42]

    J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie. Thinking in space: How multi- modal large language models see, remember, and recall spaces. InThe IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10632–10643, 2025

    work page 2025

  43. [43]

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. InThe IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455–14465, 2024

    work page 2024

  44. [44]

    M. Jia, Z. Qi, S. Zhang, W. Zhang, X. Yu, J. He, H. Wang, and L. Yi. Omnispatial: Towards comprehensive spatial reasoning benchmark for vision language models. InInternational Con- ference on Learning Representations (ICLR), 2026. 11

    work page 2026

  45. [45]

    Zhang, Z

    P. Zhang, Z. Huang, Y . Wang, J. Zhang, L. Xue, Z. Wang, Q. Wang, K. Chandrasegaran, R. Zhang, Y . Choi, et al. Theory of space: Can foundation models construct spatial beliefs through active exploration? InInternational Conference on Learning Representations (ICLR), 2026

    work page 2026

  46. [46]

    D. Wu, F. Liu, Y .-H. Hung, and Y . Duan. Spatial-mllm: Boosting mllm capabilities in visual- based spatial intelligence.Advances in Neural Information Processing Systems (NeurIPS), 38: 13569–13597, 2025

    work page 2025

  47. [47]

    R. Yang, H. Chen, J. Zhang, M. Zhao, C. Qian, K. Wang, Q. Wang, T. V . Koripella, M. Mova- hedi, M. Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. InInternational Conference on Machine Learning (ICML), pages 70576–70631. PMLR, 2025

    work page 2025

  48. [48]

    E. Zhou, J. An, C. Chi, Y . Han, S. Rong, C. Zhang, P. Wang, Z. Wang, T. Huang, L. Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. Advances in Neural Information Processing Systems (NeurIPS), 38:28404–28481, 2025

    work page 2025

  49. [49]

    UniTeam: Open Vocabulary Mobile Manipulation Challenge

    A. Melnik, M. B ¨uttner, L. Harz, L. Brown, G. C. Nandi, A. PS, G. K. Yadav, R. Kala, and R. Haschke. Uniteam: Open vocabulary mobile manipulation challenge.arXiv preprint arXiv:2312.08611, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  50. [50]

    Introducing gpt-4.1 in the api.https://openai.com/index/gpt-4-1/, 2025

    OpenAI. Introducing gpt-4.1 in the api.https://openai.com/index/gpt-4-1/, 2025

    work page 2025

  51. [51]

    Introducing gpt-5.2.https://openai.com/index/introducing-gpt-5-2/, 2025

    OpenAI. Introducing gpt-5.2.https://openai.com/index/introducing-gpt-5-2/, 2025

    work page 2025

  52. [52]

    Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3

    Qwen. Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3. 5, 2025

    work page 2025

  53. [53]

    Qwen3.6-27b: Flagship-level coding in a 27b dense model.https://qwen.ai/blog? id=qwen3.6-27b, 2025

    Qwen. Qwen3.6-27b: Flagship-level coding in a 27b dense model.https://qwen.ai/blog? id=qwen3.6-27b, 2025

    work page 2025

  54. [54]

    W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  55. [55]

    B. R. Team, M. Cao, H. Tan, Y . Ji, X. Chen, M. Lin, Z. Li, Z. Cao, P. Wang, E. Zhou, et al. Robobrain 2.0 technical report.arXiv preprint arXiv:2507.02029, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  56. [56]

    Gemini 335: Stereo vision camera for robotics.https://orbbec.com/products/ stereo-vision-camera/gemini-335/, 2026

    Orbbec. Gemini 335: Stereo vision camera for robotics.https://orbbec.com/products/ stereo-vision-camera/gemini-335/, 2026. Product page, accessed 2026-06-02

    work page 2026

  57. [57]

    T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and D. Rus. Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping. InInternational Conference on Intelligent Robots and Systems (IROS), pages 5135–5142. IEEE, 2020

    work page 2020

  58. [58]

    Macenski, F

    S. Macenski, F. Martin, R. White, and J. Gin´es Clavero. The marathon 2: A navigation system. InInternational Conference on Intelligent Robots and Systems (IROS), 2020. 12 The Appendix is organized into the following sections:Method Details(Section A),More Exper- iments & Results(Section B). A Method Details 13 A.1 View Selection . . . . . . . . . . . . ....

    work page 2020

  59. [59]

    Visibility: The{object name}is clearly visible and identifiable (not blurry, not heavily occluded, and enough of the object is visible to pick a precise grasp/target point)

    work page

  60. [60]

    Avoid images where obstacles/clutter block the approach path to the{object name}or where the{object name}appears unlikely to be reachable

    Reachability/Navigability: The robot can realistically move close enough to the{object name}to pick it up. Avoid images where obstacles/clutter block the approach path to the{object name}or where the{object name}appears unlikely to be reachable. Return a JSON object with key ‘ID’ where ‘ID’ is one of{image id list}. If you cannot find the{object name}, st...

    work page

  61. [61]

    Visibility: The surface of the{place recep name}is clearly visible (not blurry, not heavily occluded, enough surface area visible to select a stable point for placing the object)

    work page

  62. [62]

    Avoid images where obstacles/clutter block access to the front/edge of the{place recep name}or where approaching the {place recep name}would likely be impossible

    Reachability/Navigability: The robot can realistically move close enough to the{place recep name}to perform placement. Avoid images where obstacles/clutter block access to the front/edge of the{place recep name}or where approaching the {place recep name}would likely be impossible. Return a JSON object with key ‘ID’ where ‘ID’ is one of{image id list}. If ...

    work page

  63. [63]

    Visible-on-object point: The pixel must lie ON the{object name}(not background or other items), and should be on a clearly visible, unoccluded part of the object (avoid blurry/ambiguous regions and heavy occlusions)

    work page

  64. [64]

    Prefer points on the object that are not blocked by obstacles between the robot and other object

    Reachable for pickup: Choose a point such that the robot can realistically approach near and pick up the object. Prefer points on the object that are not blocked by obstacles between the robot and other object. Avoid cases where the object (or the selected point) is behind large barriers so the robot cannot get close enough to grasp. Return the pixel coor...

    work page

  65. [65]

    The point should NOT overlap/collide with any other unrelated objects

    Safe & stable surface: The point must lie on a flat, supported, and stable region of the{place recep name}. The point should NOT overlap/collide with any other unrelated objects

    work page

  66. [66]

    Do NOT select points on or near the {place recep name}’s edge

    Avoid the edge (anti-drop): Prefer an interior placement region rather than the boundary. Do NOT select points on or near the {place recep name}’s edge

    work page

  67. [67]

    Reachable placement (base + arm constraints): The point should be reachable for the robot to approach and place the object. Prefer locations with clear free space around the receptacle and no obvious obstacles blocking the robot’s approach and placing path (e.g., clutter, furniture blocking the front of the{place recep name}). Avoid points that appear beh...

    work page

  68. [68]

    Pick up the cup from the table. (Fig. 11)

    work page

  69. [69]

    Place the cake on the plate. (Fig. 12)

    work page

  70. [70]

    Place the bottle on the table, in front of the monitor. (Fig. 13)

    work page

  71. [71]

    in front of the monitor

    Imagine facing the monitor, place the cake at the bottom-left corner of the table. (Fig. 14) The first two tasks mainly evaluate whether UniLM-Nav can effectively bridge object navigation and manipulation in real scenes, where the robot needs to approach the task-relevant object or receptacle and execute the corresponding manipulation. Furthermore, the la...

    work page