Pith. sign in

REVIEW 3 major objections 5 minor 66 references

Token-wise streaming of a slow VLM's internal states lets a fast planner act on fresh guidance, lifting dynamic navigation success 10 points over wait-then-act.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:52 UTC pith:KLIGCQRL

load-bearing objection Genuinely new streaming idea and a clean ablation, but the central mechanism is not instantiable as written because the Sequence-to-Slot Latent Bridge has no training protocol and the planner never learned language goals. the 3 major comments →

arxiv 2607.16806 v1 pith:KLIGCQRL submitted 2026-07-18 cs.RO

Token-Wise Latent Streaming from Slow Reasoners to Fast Planners for Dynamic Vision Language Navigation

Tianshuai Hu , Yangyi Zhong , Zeying Gong , Lingdong Kong , Xiaodong Mei , Guoyang Zhao , Xiaolu Liu , Song Wang
show 2 more authors
Rong Li Junwei Liang
This is my paper
classification cs.RO
keywords vision-language navigationdual-system policytoken-wise streaminglatent slotsobservation stalenesssocial navigationflow matchingdynamic environments
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that in dynamic vision-language navigation the real cost is not reasoning quality but the timing of guidance: a robot that waits for a slow language model to finish its answer acts on a scene the world has already outrun. SPARK-VLN tests the hypothesis that a VLM's intermediate hidden states already encode action-relevant intent long before generation completes, and streams those states token by token into a fast flow-matching planner. On a new human-centric benchmark where pedestrians keep moving throughout inference, the framework reaches 34.8% SR and 28.68% SPL in the realistic setting and outperforms a blocking dual-system baseline by 10.0 SR points, while cutting per-update latency from 0.788s to 0.185s. The claim, if correct, means language-grounded planning signals can be delivered progressively without paying the full latency of slow reasoning, and that observation staleness — not model scale — is the decisive factor in dynamic social navigation.

Core claim

SPARK-VLN's central claim is that autoregressive VLM inference can be treated as a streaming source rather than a blocking call. By extracting hidden states from designated decoder layers at every generated token, compressing the growing sequence into a fixed set of N=8 latent slots through cross-attention with learnable queries, and infusing those slots into a rectified-flow planner alongside current RGB-D observations, the planner receives progressively refined language-grounded intent while the VLM is still reasoning. Measured on the proposed benchmark's Realistic Dynamic Environment — where the simulator runs continuously — this yields 34.8% SR, 28.68% SPL, 93.9% PSC and 29.3% collision

What carries the argument

Three coupled modules carry the argument. The Token-Wise Hidden Streamer extracts hidden states from a designated subset of VLM decoder layers at every generated token, producing a growing tensor. The Sequence-to-Slot Latent Bridge adds layer and time positional encodings, projects the tensor into the planner's dimension, and compresses it into N=8 latent slots via multi-head cross-attention with learnable queries, decoupling the planner's fixed-size conditioning from the VLM's variable output length. The Evolving Latent Conditioner fuses the up-to-date slots with current egocentric RGB-D observation tokens as the planner conditioning, and a rectified flow-matching trajectory generator with

Load-bearing premise

The central bet is that the frozen VLM's partial hidden states, after projection through an underspecified latent bridge, convey enough goal direction and social context to a planner that was trained only on point-goal navigation and is given no goal coordinate during language evaluation; if the bridge is not actually transmitting language intent, the reported improvements could stem from unstated supervision or from the slots leaking the current egocentric observation rather

What would settle it

Feed the planner latent slots computed from a VLM that received a scrambled or empty instruction, keeping the visual input and all other components identical; if the success rate does not fall toward the wait-then-act baseline of 24.8% SR, then the slots carry scene information rather than instruction semantics, and the streaming gain is not evidence of progressive reasoning.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dual-system architectures that hand over guidance only after slow reasoning finishes can be upgraded to token-wise streaming without changing either the reasoner or the planner, recovering the same reactivity gains reported here.
  • Latency-aware evaluation becomes feasible: the gap between the Idealized and Realistic Dynamic Environments measures staleness robustness, and future benchmarks can report pedestrian displacement during inference as a standard safety statistic.
  • Because the planner is pretrained on point-goal data and language enters only through the streamed slots, the same planner can be steered by new instruction modalities by retraining the latent bridge alone.
  • Streaming shortens the effective control period: with average latency at 0.185s, pedestrians move only 0.050m during one inference step, which directly reduces collision risk in crowded scenes.
  • The finding generalizes to any closed-loop system with a slow multimodal reasoner and a fast actuator, suggesting token-wise conditioning as a drop-in pattern for latency-critical vision-language-action policies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the streaming gain is genuinely about reasoning progress, performance should improve monotonically as more tokens stream; a flat performance from the first token onward would suggest the slots are carrying static visual context rather than evolving language intent — a distinction the paper does not test.
  • The Sequence-to-Slot Latent Bridge is the only language-conditioned component not documented with a training loss or dataset; reproducing the result requires discovering that protocol, and the claim's generality depends on whether the bridge can be trained on point-goal data alone or needs explicit VLN supervision.
  • The latent slots likely function as a soft attention mask that weights planner features toward instruction-relevant scene regions; probing them by decoding or by ablating slots tied to direction words could turn the slots into an interpretable interface between language and control.
  • Streaming the reasoner's hidden states is complementary to predicting pedestrian futures: coupling token-wise guidance with anticipatory social forecasting could push the planner from reactive to predictive compliance, an extension the paper names as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SPARK-VLN, a dual-system vision-language navigation framework in which a slow VLM reasoner (VILA-8B) streams its intermediate hidden states token-by-token to a fast flow-matching expert planner. Three modules are introduced: a Token-Wise Hidden Streamer, a Sequence-to-Slot Latent Bridge, and an Evolving Latent Conditioner. The authors argue that this token-wise streaming alleviates observation staleness in dynamic social environments and demonstrate it on a newly introduced asynchronous dynamic-social VLN benchmark, reporting 34.80% SR / 28.68% SPL in the Realistic Dynamic Environment (Table I), a +10.0 SR improvement over Wait-then-Act (Table III), and reduced per-update latency (Table V).

Significance. The core idea is timely and plausible: autoregressive VLMs produce usable intermediate representations, and delivering them to a fast planner before generation completes is a natural way to reduce observation staleness in reactive social navigation. The architecture is clearly motivated, and the central ablation (Stream vs. Wait-then-Act) holds the VLM and planner fixed, which is the right control for the main claim. If the mechanism works as described, the contribution is significant for latency-aware VLA and social VLN. However, the paper as written cannot be instantiated from its stated components: the Sequence-to-Slot Latent Bridge has no training protocol, and the quantitative evaluation lacks error bars and a released benchmark. The manuscript is internally consistent in its equations, but the load-bearing premise that streamed slots encode language-derived intent into a point-goal-trained planner is asserted rather than demonstrated.

major comments (3)
  1. [IV-B, V-A, Eq. (14)] The Sequence-to-Slot Latent Bridge is defined only architecturally: learnable W_p, positional encodings e_layer/e_time, N=8 queries Q, cross-attention, and an FFN (Eqs. 9–12). No training loss, dataset, optimizer, or protocol is given for these parameters. This is load-bearing because Eq. (14) sets the language-goal conditioning to R_t, replacing the point-goal vector W_g*g, while Section V-A states that the expert planner is trained on point-goal navigation only and the VLM is frozen. The paper therefore does not explain how R_t acquires language-goal semantics that can condition a planner that never saw language goals. Please specify the bridge training objective (e.g., end-to-end flow-matching loss through Eq. (16), imitation against point-goal goal vectors, or a separate alignment loss), the data used, and whether gradients flow into the bridge during training. Without this, the repo
  2. [Tables I and II] The main quantitative comparisons report no standard errors, confidence intervals, number of episodes, or seeds. The benchmark is introduced in this paper and not released with the manuscript, so the differences that support the central claims—e.g., 4.4 SR over NaVILA in the Idealized environment, 3.0 PSC in the Realistic environment—may be within run-to-run noise. Please report mean±std over at least three seeds, state the number of instruction episodes evaluated, and make the benchmark (or a detailed evaluation protocol) publicly available for external verification.
  3. [Table V and runtime analysis] The claimed latency reduction from 0.788 s (W-t-A) to 0.185 s (Stream) compares two different quantities. The caption labels both as 'per-inference latency,' but W-t-A's value is the time for a full VLM forward followed by one planner update, whereas Stream's value is the interval between planner updates during the same VLM forward. Under streaming, the planner updates multiple times per inference, so the end-to-end observation-to-final-action latency is not necessarily cut; only the inter-update interval is. The paper should report the total decision latency for one instruction-following cycle, the number of planner updates per cycle, and clarify whether the pedestrian displacement values (0.213 m vs. 0.050 m average) are per inference cycle or per planner update.
minor comments (5)
  1. [Abstract] Typo: 'mproves' should be 'improves'.
  2. [Eq. (3)] Notation is ambiguous: f_vlm maps x_{t'} to z_{t'}, and c_t = phi(z_{t'}) but c_t is indexed by t while z is indexed by t'. Please clarify the time indices and the order of projection/infusion.
  3. [Eq. (14)] z_g^{(m)} is a vector for point-goal (W_g g) but a matrix R_t of size N×d for language-goal. The paper should state how c_t = [z_obs; z_g^{(m)}] concatenates a matrix with z_obs, and confirm the planner architecture accepts this variable-shape conditioning.
  4. [IV-B] The phrase 'factorized spatio-temporal positional encodings' is used for encodings over layer index and decoding step. This is not spatial or temporal in the usual sense, and the terminology is confusing. Consider renaming to 'layer and time-step positional encodings.'
  5. [Table V] The 'Max' columns contain very large values (e.g., Uni-NaVid Max latency 0.795 s vs. Avg 0.193 s; DualVLN Max displacement 2.801 m). Clarify whether the maximum is across episodes, across planning updates, or across pedestrian trajectories, and describe what these outliers represent.

Circularity Check

0 steps flagged

No significant circularity; the main gap is an unstated bridge-training protocol, which is a reproducibility issue, not a circular derivation.

full rationale

The claimed derivation chain is structurally non-circular. The method defines (i) a streamer that collects per-token multi-layer hidden states H (Eq. 7-8), (ii) a bridge that projects and compresses them into latent slots R_t via learnable queries and cross-attention (Eq. 9-12), (iii) a conditioner that feeds R_t as the language-goal token z_g^(m) (Eq. 14), and (iv) a flow-matching planner optimizing L_RF (Eq. 16). None of these definitions presupposes the outcome it is used to explain: R_t is not defined as the SR gain or as the W-t-A difference; it is an independently specified transformation of VLM hidden states. The central empirical claim — that token-wise streaming outperforms wait-then-act — is tested in Table III with the same VLM and planner, so the +10.0 SR is a measured outcome, not a fitted input. The new benchmark could in principle reward any method; baselines are evaluated under the same protocol, so there is no self-fulfilling metric. The authors do cite their own prior work ([6]-[8]) for the general need for reactive social navigation, but these citations are background only; the load-bearing design choices (Perceiver-style cross-attention [58], rectified flow [59]-[61], VILA backbone [11]) are external. The genuine weakness is that the Sequence-to-Slot Latent Bridge is described only architecturally (Sec. IV-B) with no training loss, data, or protocol, and the planner is said to be trained on point-goal navigation only (Sec. V-A); as written, the VLN-conditioned planner is under-specified. That is an omitted-proof / reproducibility problem, not a circularity, because no equation or fitted parameter is silently reused as its own prediction.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central result depends on several unverified assumptions: partial hidden states carry actionable intent, a Perceiver-style bottleneck preserves that intent, a point-goal-trained planner can be driven by language through slots, and the new ORCA-based simulator captures real dynamic social navigation. None of these is proven independently; all are necessary for the claimed improvements.

free parameters (5)
  • Slot bridge learnables (queries Q, projection W_p, positional encodings e_layer/e_time, cross-attention/FFN) = not reported
    Learned during training; no training data, loss, or optimizer details are given.
  • Flow-matching planner and critic weights = not reported
    Trained on point-goal navigation; weights are not released.
  • N=8 latent slots = 8
    Hand-chosen bottleneck size; no sensitivity study is reported.
  • Layer subset L={16,...,23} = 16-23
    Hand-chosen extraction layers; no justification or search is provided.
  • Euler integration steps J=5 = 5
    Hand-chosen for the latency-accuracy tradeoff; no ablation on J is reported.
axioms (5)
  • domain assumption Intermediate hidden states of a frozen VLM encode action-relevant intent before generation completes.
    Section I states this as the key observation; it is the motivating premise but is not directly measured or compared against alternative signals.
  • domain assumption A Perceiver-style cross-attention bottleneck with N=8 learnable queries preserves planner-relevant information from the variable-length hidden stream.
    Section IV-B, Eq. (11); no theoretical guarantee, only an empirical ablation against mean pooling.
  • domain assumption A point-goal-trained planner can be driven by language instruction purely through streamed latent slots when the goal coordinate is zeroed.
    V-A Implementation Details and Eq. (14); the paper does not provide training evidence for this transfer.
  • domain assumption Simulated ORCA-driven pedestrians in the new benchmark capture safety-critical dynamic social navigation.
    Section V-A; no real-world validation or comparison to established social navigation benchmarks is provided.
  • standard math Rectified flow matching can model multimodal trajectories for navigation.
    Eqs. (13)-(17); this is a standard generative modeling assumption imported from [59,60].
invented entities (1)
  • Latent slot guidance R_t no independent evidence
    purpose: Fixed-size communication channel from the VLM hidden stream to the fast planner
    An internal learned representation; the paper does not show that individual slots decode to specific semantic content, and there is no falsifiable handle outside the pipeline.

pith-pipeline@v1.3.0-alltime-deepseek · 18073 in / 12168 out tokens · 125247 ms · 2026-08-01T19:52:20.352452+00:00 · methodology

0 comments
read the original abstract

Vision-Language Navigation in dynamic, human-centric environments exposes a fundamental tension: linguistic reasoning is slow and deliberative, whereas safe, socially compliant planning should be instant and reactive. The resulting observation staleness is safety-critical: a maneuver chosen during inference can already be unsafe by the time it executes. We observe that, long before a VLM finishes its inference, its intermediate hidden states already encode action-relevant intent. We propose SPARK-VLN, a dual-system framework for dynamic social VLN that streams the slow VLM reasoner's knowledge to a fast flow-matching expert planner throughout token generation, providing fresh and evolving guidance during inference. This design is realized by three modules: a Token-Wise Hidden Streamer that extracts intermediate hidden states along the token generation process, a Sequence-to-Slot Latent Bridge that projects them into fixed-size latent slots, and an Evolving Latent Conditioner that infuses them into the expert planner. We also introduce a human-centric benchmark suite for dynamic social vision-language navigation that keeps pedestrians and the robot active throughout inference and reports navigation success, social compliance, human collisions, and explicit staleness statistics. Across these settings, SPARK-VLN mproves navigation success and social compliance while sustaining inference efficiency. Webpage: https://hutslib.github.io/SPARK-VLN/.

Figures

Figures reproduced from arXiv: 2607.16806 by Guoyang Zhao, Junwei Liang, Lingdong Kong, Rong Li, Song Wang, Tianshuai Hu, Xiaodong Mei, Xiaolu Liu, Yangyi Zhong, Zeying Gong.

Figure 1
Figure 1. Figure 1: Comparison of VLN execution paradigms. (a) Single￾System (Reason-Then-Act): the VLM must finish full inference before emitting an action; the agent idles while the environment advances. (b) Dual-System (Blocking Update): the expert planner acts continuously, but receives guidance (yellow arrows) only after the VLM finishes, yielding a stale conditioning signal. (c) Ours (Token Streaming): SPARK-VLN streams… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the SPARK-VLN framework. Top: The VLM backbone processes egocentric RGB and instruction tokens; the Token-Wise Hidden Streamer extracts hidden states from designated layers during autoregressive generation, producing an evolving stream Hkt ∈ R l×kt×D. Middle: The Sequence-to-Slot Latent Bridge augments the stream with layer and time positional encodings, projects it into the planner’s embedding… view at source ↗
Figure 3
Figure 3. Figure 3: Data flow of SPARK-VLN. The slow VLM reasoner receives the language instruction together with 8 recent egocentric RGB frames. As it autoregressively generates the answer (e.g., “The next action is move forward 75 cm.”), the Token-Wise Hidden Streamer extracts intermediate-layer hidden states during generation, and the Sequence-to-Slot Latent Bridge compresses the evolving hidden stream into N=8 fixed-size … view at source ↗
Figure 4
Figure 4. Figure 4: Overview of our dynamic social VLN benchmark. (a) Benchmark Overview. The benchmark spans diverse indoor scenes (such as bedroom, office, hall, and corridor) and covers several human encounter patterns, including frontal approach, intersection, following, and corner, with the robot path in green and the pedestrian path in red. It supports both the PointNav and VLN tasks under two evaluation modes, the Idea… view at source ↗
Figure 5
Figure 5. Figure 5: Radar comparison across tasks and dynamic environments. (a,b) Vision-Language Navigation (VLN) and (c,d) point-goal navigation, each evaluated under the Idealized and Realistic dynamic environments. SPARK-VLN achieves the largest coverage in each setting, with annotated raw values. The performance gap between the Idealized and Realistic panels, i.e., (a) vs. (b) and (c) vs. (d), reflects each method’s robu… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison in dynamic environments. Each row shows the egocentric RGB view, the third-person view, and the top-down occupancy map, where the blue curve is the executed trajectory toward the goal (red pin) and the orange curve is the reference ground-truth path (computed without pedestrians). All episodes are run in the Realistic Dynamic Environment, where the simulator keeps running during infe… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison on a human-populated navigation episode. Given the same instruction (“Walk towards the desk in the office area. Stop next to the desk.”), we compare the Wait-then-Act baseline (Left) with our SPARK-VLN (Right). Each row shows the egocentric RGB view, the third-person view, and the top-down occupancy map, where the blue curve is the executed trajectory toward the goal (red pin) and th… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 25 linked inside Pith

  1. [1]

    NaVid: Video-based VLM plans the next step for vision-and-language navigation,

    J. Zhang, K. Wang, R. Xu, G. Zhou, Y . Hong, X. Fang, Q. Wu, Z. Zhang, and H. Wang, “NaVid: Video-based VLM plans the next step for vision-and-language navigation,”arXiv preprint arXiv:2402.15852, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    NaVILA: Legged robot vision- language-action model for navigation,

    A.-C. Cheng, Y . Ji, Z. Yang, Z. Gongye, X. Zou, J. Kautz, E. Bıyık, H. Yin, S. Liu, and X. Wang, “NaVILA: Legged robot vision- language-action model for navigation,” inRobot. Sci. Syst., 2025

    2025

  3. [3]

    Uni-NaVid: A video-based vision-language- action model for unifying embodied navigation tasks,

    J. Zhang, K. Wang, S. Wang, M. Li, H. Liu, S. Wei, Z. Wang, Z. Zhang, and H. Wang, “Uni-NaVid: A video-based vision-language- action model for unifying embodied navigation tasks,”arXiv preprint arXiv:2412.06224, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    StreamVLN: Streaming vision-and- language navigation via SlowFast context modeling,

    M. Wei, C. Wan, X. Yu, T. Wang, Y . Yang, X. Mao, C. Zhu, W. Cai, H. Wang, Y . Chenet al., “StreamVLN: Streaming vision-and- language navigation via SlowFast context modeling,”arXiv preprint arXiv:2507.05240, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Ground slow, move fast: A dual-system foundation model for generalizable vision-language navigation,

    M. Wei, C. Wan, J. Peng, X. Yu, Y . Yang, D. Feng, W. Cai, C. Zhu, T. Wang, J. Pang, and X. Liu, “Ground slow, move fast: A dual-system foundation model for generalizable vision-language navigation,” inInt. Conf. Learn. Represent., 2026

    2026

  6. [6]

    NavThinker: Action-conditioned world models for coupled prediction and planning in social navigation,

    T. Hu, Z. Gong, L. Kong, X. Mei, Y . Ding, Q. Zeng, A. Liang, R. Li, Y . Zhong, and J. Liang, “NavThinker: Action-conditioned world models for coupled prediction and planning in social navigation,” arXiv preprint arXiv:2603.15359, 2026

    arXiv 2026

  7. [7]

    FLUX: Accelerating cross-embodiment generative navigation policies via rectified flow and static-to-dynamic learning,

    Z. Gong, Y . Zhong, Y . Ding, T. Hu, G. Zhao, L. Kong, R. Li, J. You, and J. Liang, “FLUX: Accelerating cross-embodiment generative navigation policies via rectified flow and static-to-dynamic learning,” arXiv preprint arXiv:2603.12806, 2026

    arXiv 2026

  8. [8]

    From cognition to precognition: A future-aware framework for social navigation,

    Z. Gong, T. Hu, R. Qiu, and J. Liang, “From cognition to precognition: A future-aware framework for social navigation,” inIEEE Int. Conf. Robot. Autom., 2025, pp. 9122–9129

    2025

  9. [9]

    Qwen2.5-VL technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-VL technical report,”arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    LLaMA-VID: An image is worth 2 tokens in large language models,

    Y . Li, C. Wang, and J. Jia, “LLaMA-VID: An image is worth 2 tokens in large language models,” inEur. Conf. Comput. Vis.Springer, 2024, pp. 323–340

    2024

  11. [11]

    VILA: On pre-training for visual language models,

    J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han, “VILA: On pre-training for visual language models,” inIEEE/CVF Conf. Comput. Vis. Pattern Recog., 2024, pp. 26 689–26 699

    2024

  12. [12]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman et al., “GPT-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y . Qiao, and J. Dai, “InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” inIEEE/CVF Conf. Comput. Vis. Pattern Recog., 2024, pp. 24 185–24 198

    2024

  14. [14]

    Real-time execution of action chunking flow policies,

    K. Black, M. Galliker, and S. Levine, “Real-time execution of action chunking flow policies,” inAdv. Neural Inf. Process. Syst., vol. 38, 2025, pp. 33 383–33 407

    2025

  15. [15]

    Training-time ac- tion conditioning for efficient real-time chunking,

    K. Black, A. Z. Ren, M. Equi, and S. Levine, “Training-time ac- tion conditioning for efficient real-time chunking,”arXiv preprint arXiv:2512.05964, 2025

    arXiv 2025

  16. [16]

    DynamicVLA: A vision-language-action model for dynamic object manipulation,

    H. Xie, B. Wen, J. Zheng, Z. Chen, F. Hong, H. Diao, and Z. Liu, “DynamicVLA: A vision-language-action model for dynamic object manipulation,”arXiv preprint arXiv:2601.22153, 2026

    arXiv 2026

  17. [17]

    VLASH: Real-time VLAs via future-state-aware asynchronous inference,

    J. Tang, Y . Sun, Y . Zhao, S. Yang, Y . Lin, Z. Zhang, J. Hou, Y . Lu, Z. Liu, and S. Han, “VLASH: Real-time VLAs via future-state-aware asynchronous inference,”arXiv preprint arXiv:2512.01031, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Faster: Rethinking real-time flow VLAs,

    Y . Lu, Z. Liu, X. Fan, Z. Yang, J. Hou, J. Li, K. Ding, and H. Zhao, “Faster: Rethinking real-time flow VLAs,”arXiv preprint arXiv:2603.19199, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    TIC-VLA: A think-in-control vision-language-action model for robot navigation in dynamic environments,

    Z. Huang, Y . Zhang, J. Liu, R. Song, C. Tang, and J. Ma, “TIC-VLA: A think-in-control vision-language-action model for robot navigation in dynamic environments,”arXiv preprint arXiv:2602.02459, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S ¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” inIEEE/CVF Conf. Comput. Vis. Pattern Recog., 2018, pp. 3674–3683

    2018

  21. [21]

    Room- across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,

    A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge, “Room- across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding,” inProc. Conf. Empir. Methods Nat. Lang. Process., 2020, pp. 4392–4412

    2020

  22. [22]

    Beyond the nav-graph: Vision-and-language navigation in continuous environ- ments,

    J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous environ- ments,” inEur. Conf. Comput. Vis.Springer, 2020, pp. 104–120

    2020

  23. [23]

    Reciprocal n-body collision avoidance,

    J. van den Berg, S. J. Guy, M. Lin, and D. Manocha, “Reciprocal n-body collision avoidance,” inInt. Sym. Robot. Research. Springer, 2011, vol. 70, pp. 3–19

    2011

  24. [24]

    Speaker- follower models for vision-and-language navigation,

    D. Fried, R. Hu, V . Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell, “Speaker- follower models for vision-and-language navigation,” inAdv. Neural Inf. Process. Syst., vol. 31, 2018

    2018

  25. [25]

    VLN BERT: A recurrent vision-and-language BERT for navigation,

    Y . Hong, Q. Wu, Y . Qi, C. Rodriguez-Opazo, and S. Gould, “VLN BERT: A recurrent vision-and-language BERT for navigation,” in IEEE/CVF Conf. Comput. Vis. Pattern Recog., 2021, pp. 1643–1653

    2021

  26. [26]

    Qwen3-VL technical report,

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Geet al., “Qwen3-VL technical report,”arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    MapNav: A novel memory repre- sentation via annotated semantic maps for VLM-based vision-and- language navigation,

    L. Zhang, X. Hao, Q. Xu, Q. Zhang, X. Zhang, P. Wang, J. Zhang, Z. Wang, S. Zhang, and R. Xu, “MapNav: A novel memory repre- sentation via annotated semantic maps for VLM-based vision-and- language navigation,” inAnnu. Meet. Assoc. Comput. Linguist., 2025, pp. 13 032–13 056

    2025

  28. [28]

    JanusVLN: Decoupling semantics and spatiality with dual implicit memory for vision-language navigation,

    S. Zeng, D. Qi, X. Chang, F. Xiong, S. Xie, X. Wu, S. Liang, M. Xu, X. Wei, and N. Guo, “JanusVLN: Decoupling semantics and spatiality with dual implicit memory for vision-language navigation,”arXiv preprint arXiv:2509.22548, 2025

    arXiv 2025

  29. [29]

    Embodied navigation foundation model,

    J. Zhang, A. Li, Y . Qi, M. Li, J. Liu, S. Wang, H. Liu, G. Zhou, Y . Wu, X. Liet al., “Embodied navigation foundation model,”arXiv preprint arXiv:2509.12129, 2025

    arXiv 2025

  30. [30]

    GN0: Toward a unified paradigm for generation, evaluation, and policy learning in visual-language navigation,

    X. Li, X. Zhang, Y . Huang, J. Dong, T. Wang, S. Zhou, Y . Wu, C. Sun, Y . Ge, Q. Wenget al., “GN0: Toward a unified paradigm for generation, evaluation, and policy learning in visual-language navigation,”arXiv preprint arXiv:2606.03682, 2026

    Pith/arXiv arXiv 2026

  31. [31]

    Towards unified world models for visual navigation via memory-augmented planning and foresight,

    Y . Dong, F. Wu, G. Chen, L. Kong, X. Zhu, Q. Hu, Y . Zhou, J. Sun, J.- Y . He, Q. Dai, A. G. Hauptmann, and Z.-Q. Cheng, “Towards unified world models for visual navigation via memory-augmented planning and foresight,”arXiv preprint arXiv:2510.08713, 2025

    arXiv 2025

  32. [32]

    Language- conditioned world modeling for visual navigation,

    Y . Dong, F. Wu, Y . Dai, L. Kong, G. Chen, X. Zhu, Q. Hu, T. Wang, J. Garnica, F. Liu, S. Huang, Q. Dai, and Z.-Q. Cheng, “Language- conditioned world modeling for visual navigation,”arXiv preprint arXiv:2603.26741, 2026

    arXiv 2026

  33. [33]

    3D and 4D world modeling: A survey,

    L. Kong, W. Yang, J. Mei, Y . Liu, A. Liang, D. Zhu, D. Lu, W. Yin, X. Hu, M. Jia, J. Deng, K. Zhang, Y . Wu, T. Yan, S. Gao, S. Wang, L. Li, L. Pan, Y . Liu, J. Zhu, W. T. Ooi, S. C. H. Hoi, and Z. Liu, “3D and 4D world modeling: A survey,”arXiv preprint arXiv:2509.07996, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Stairway to success: An online floor-aware zero- shot object-goal navigation framework via LLM-driven coarse-to-fine exploration,

    Z. Gong, R. Li, T. Hu, R. Qiu, L. Kong, L. Zhang, G. Zhao, Y . Ding, and J. Liang, “Stairway to success: An online floor-aware zero- shot object-goal navigation framework via LLM-driven coarse-to-fine exploration,”IEEE Robot. Autom. Lett., 2025

    2025

  35. [35]

    Vision-language-action models for autonomous driving: Past, present, and future,

    T. Hu, X. Liu, S. Wang, Y . Zhu, A. Liang, L. Kong, G. Zhao, Z. Gong, J. Cen, Z. Huanget al., “Vision-language-action models for autonomous driving: Past, present, and future,”arXiv preprint arXiv:2512.16760, 2025

    arXiv 2025

  36. [36]

    Structured observation language for efficient and generalizable vision-language navigation,

    D. Peng, F. Ma, and J. Ma, “Structured observation language for efficient and generalizable vision-language navigation,”arXiv preprint arXiv:2603.27577, 2026

    Pith/arXiv arXiv 2026

  37. [37]

    AR-VLA: True autoregres- sive action expert for vision-language-action models,

    Y . Hu, J.-N. Zaech, N. Nikolov, Y . Yao, S. Dey, G. Albanese, R. Detry, L. Van Gool, and D. Paudel, “AR-VLA: True autoregres- sive action expert for vision-language-action models,”arXiv preprint arXiv:2603.10126, 2026

    Pith/arXiv arXiv 2026

  38. [38]

    Leave no observation behind: Real-time correction for VLA action chunks,

    K. Sendai, M. Alvarez, T. Matsushima, Y . Matsuo, and Y . Iwasawa, “Leave no observation behind: Real-time correction for VLA action chunks,”arXiv preprint arXiv:2509.23224, 2025

    arXiv 2025

  39. [39]

    Language-conditioned robotic manipulation with fast and slow thinking,

    M. Zhu, Y . Zhu, J. Li, J. Wen, Z. Xu, Z. Che, C. Shen, Y . Peng, D. Liu, F. Feng, and J. Tang, “Language-conditioned robotic manipulation with fast and slow thinking,” inIEEE Int. Conf. Robot. Autom., 2024

    2024

  40. [40]

    HiRT: Enhancing robotic control with hierarchical robot transform- ers,

    J. Zhang, Y . Guo, X. Chen, Y .-J. Wang, Y . Hu, C. Shi, and J. Chen, “HiRT: Enhancing robotic control with hierarchical robot transform- ers,” inConf. Robot Learn., 2024

    2024

  41. [41]

    A dual process VLA: Efficient robotic manipulation leveraging VLM,

    B. Han, J. Kim, and J. Jang, “A dual process VLA: Efficient robotic manipulation leveraging VLM,”arXiv preprint arXiv:2410.15549, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    A self-correcting vision-language- action model for fast and slow system manipulation,

    C. Li, J. Liu, G. Wang, X. Li, S. Chen, L. Heng, C. Xiong, J. Ge, R. Zhang, K. Zhou, and S. Zhang, “A self-correcting vision-language- action model for fast and slow system manipulation,”arXiv preprint arXiv:2405.17418, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    OpenHelix: A short survey, empirical analysis, and open-source dual-system VLA model for robotic manipulation,

    C. Cui, P. Ding, W. Song, S. Bai, X. Tong, Z. Ge, R. Suo, W. Zhou, Y . Liu, B. Jia, H. Zhao, S. Huang, and D. Wang, “OpenHelix: A short survey, empirical analysis, and open-source dual-system VLA model for robotic manipulation,”arXiv preprint arXiv:2505.03912, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    OneTwoVLA: A unified vision-language-action model with adaptive reasoning,

    F. Lin, R. Nai, Y . Hu, J. You, J. Zhao, and Y . Gao, “OneTwoVLA: A unified vision-language-action model with adaptive reasoning,”arXiv preprint arXiv:2505.11917, 2025

    arXiv 2025

  45. [45]

    Hume: Introducing system-2 thinking in visual-language-action model,

    H. Song, D. Qu, Y . Yao, Q. Chen, Q. Lv, Y . Tang, M. Shi, G. Ren, M. Yao, B. Zhao, D. Wang, and X. Li, “Hume: Introducing system-2 thinking in visual-language-action model,”arXiv preprint arXiv:2505.21432, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Fast-in-slow: A dual-system foundation model unifying fast manipulation within slow reasoning,

    H. Chen, J. Liu, C. Gu, Z. Liu, R. Zhang, X. Li, X. He, Y . Guo, C.-W. Fu, S. Zhang, and P.-A. Heng, “Fast-in-slow: A dual-system foundation model unifying fast manipulation within slow reasoning,” arXiv preprint arXiv:2506.01953, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    RationalVLA: A rational vision-language-action model with dual system,

    W. Song, J. Chen, W. Li, X. He, H. Zhao, C. Cui, P. Ding, S. Su, F. Tang, X. Cheng, D. Wang, Z. Ge, X. Zheng, Z. Liu, H. Wang, and H. Li, “RationalVLA: A rational vision-language-action model with dual system,”arXiv preprint arXiv:2506.10826, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    ThinkAct: Vision-language-action reasoning via reinforced visual latent planning,

    C.-P. Huang, Y .-H. Wu, M.-H. Chen, Y .-C. F. Wang, and F.-E. Yang, “ThinkAct: Vision-language-action reasoning via reinforced visual latent planning,” inAdv. Neural Inf. Process. Syst., 2025

    2025

  49. [49]

    Xiaomi OneVL: One-step latent reasoning and planning with vision-language explanation,

    J. Lu, J. Guan, Z. Huang, J. Li, G. Li, L. Konget al., “Xiaomi OneVL: One-step latent reasoning and planning with vision-language explanation,”arXiv preprint arXiv:2604.18486, 2026

    Pith/arXiv arXiv 2026

  50. [50]

    Soc- NavBench: A grounded simulation testing framework for evaluating social navigation,

    A. Biswas, A. Wang, G. Silvera, A. Steinfeld, and H. Admoni, “Soc- NavBench: A grounded simulation testing framework for evaluating social navigation,”ACM Trans. Hum.-Robot Interact., vol. 11, no. 3, pp. 1–24, 2022

    2022

  51. [51]

    HabiCrowd: A high-performance simulator for crowd- aware visual navigation,

    A. Vuong, T. Nguyen, M. N. Vu, B. Huang, H. T. T. Binh, T. V o, and A. Nguyen, “HabiCrowd: A high-performance simulator for crowd- aware visual navigation,” inIEEE/RSJ Int. Conf. Intell. Robots Syst., 2024, pp. 5821–5827

    2024

  52. [52]

    SACSoN: Scal- able autonomous control for social navigation,

    N. Hirose, D. Shah, A. Sridhar, and S. Levine, “SACSoN: Scal- able autonomous control for social navigation,”arXiv preprint arXiv:2306.01874, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Habitat 3.0: A co-habitat for humans, avatars and robots,

    X. Puig, E. Undersander, A. Szot, M. D. Cote, T.-Y . Yanget al., “Habitat 3.0: A co-habitat for humans, avatars and robots,”arXiv preprint arXiv:2310.13724, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    Hu- NavSim: A ROS 2 human navigation simulator for benchmarking human-aware robot navigation,

    N. P ´erez-Higueras, R. Otero, F. Caballero, and L. Merino, “Hu- NavSim: A ROS 2 human navigation simulator for benchmarking human-aware robot navigation,”IEEE Robot. Autom. Lett., 2023

    2023

  55. [55]

    HA-VLN 2.0: An open benchmark and leaderboard for human-aware navigation in discrete and continuous environments with dynamic multi-human interactions,

    Y . Dong, F. Wu, Q. He, L. Kong, H. Li, M. Li, Z. Cheng, Y . Zhou, J. Sun, Q. Daiet al., “HA-VLN 2.0: An open benchmark and leaderboard for human-aware navigation in discrete and continuous environments with dynamic multi-human interactions,”arXiv preprint arXiv:2503.14229, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    LISN: Language-instructed social navigation with VLM- based controller modulating,

    J. Chen, Y . Li, P. Jiang, J. Du, Z. Chen, C. Tie, J. Deng, and L. Shao, “LISN: Language-instructed social navigation with VLM- based controller modulating,”arXiv preprint arXiv:2512.09920, 2025

    arXiv 2025

  57. [57]

    FreeAskWorld: An interactive and closed-loop simulator for human-centric embodied AI,

    Y . Peng, Y . Pan, X. He, J. Yang, X. Yin, H. Wang, X. Zheng, C. Gao, and J. Gong, “FreeAskWorld: An interactive and closed-loop simulator for human-centric embodied AI,” inAAAI Conf. Artif. Intell., 2026

    2026

  58. [58]

    Perceiver: General perception with iterative attention,

    A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira, “Perceiver: General perception with iterative attention,” inInt. Conf. Mach. Learn., 2021

    2021

  59. [59]

    Flow straight and fast: Learning to generate and transfer data with rectified flow,

    X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” inInt. Conf. Learn. Represent., 2023

    2023

  60. [60]

    Flow matching for generative modeling,

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” inInt. Conf. Learn. Represent., 2023

    2023

  61. [61]

    π0: A vision-language-action flow model for general robot control,

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finnet al., “π0: A vision-language-action flow model for general robot control,” inRobot. Sci. Syst., 2025

    2025

  62. [62]

    Diffusion policy: Visuomotor policy learning via action diffusion,

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” inRobot. Sci. Syst., 2023

    2023

  63. [63]

    iPlanner: Imperative path planning,

    F. Yang, C. Wang, C. Cadena, and M. Hutter, “iPlanner: Imperative path planning,”arXiv preprint arXiv:2302.11434, 2023

    arXiv 2023

  64. [64]

    NavDP: Learning sim-to-real navigation diffusion policy with privileged information guidance,

    W. Cai, J. Peng, Y . Yang, Y . Zhang, M. Wei, H. Wang, Y . Chen, T. Wang, and J. Pang, “NavDP: Learning sim-to-real navigation diffusion policy with privileged information guidance,”arXiv preprint arXiv:2505.08712, 2025

    arXiv 2025

  65. [65]

    On evaluation of embodied navigation agents,

    P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V . Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savvaet al., “On evaluation of embodied navigation agents,”arXiv preprint arXiv:1807.06757, 2018

    Pith/arXiv arXiv 2018

  66. [66]

    Princi- ples and guidelines for evaluating social robot navigation algorithms,

    A. Francis, C. P ´erez-D’Arpino, C. Li, F. Xia, A. Alahiet al., “Princi- ples and guidelines for evaluating social robot navigation algorithms,” arXiv preprint arXiv:2306.16740, 2023

    Pith/arXiv arXiv 2023