Pith. sign in

REVIEW 3 major objections 6 minor 52 references

Action supervision reshapes a VLA model's inherited representations, and Action QFormer shows that a query interface can make that reshaping helpful instead of disruptive.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:29 UTC pith:5EJ4AAG5

load-bearing objection A genuinely interesting VLA interface idea with a plausible mechanism story, but the central gradient-routing claim is underdetermined by a missing capacity-matched control, and the real-scene eval is too thin to carry the headline numbers. the 3 major comments →

arxiv 2607.14635 v1 pith:5EJ4AAG5 submitted 2026-07-16 cs.AI cs.CVcs.RO

Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models

Yufeng Ji , Wenhao Tang , Haoyi Niu , Koushil Sreenath , Yi Wu , Zhongyu Li This is my paper
classification cs.AI cs.CVcs.RO
keywords vision-language-action modelsrepresentation shapingaction supervisionquery-based interfacesim-to-real navigationinstruction groundinggradient routingzero-shot transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that when a vision-language-action model is finetuned for control, the action-loss gradient is not just a training signal—it physically rewrites the multimodal representations inherited from pretraining. That rewriting is necessary: without it, the model never forms clear left-right or target-directed distinctions. But applied directly to the shared backbone, it also destabilizes the representations that keep instruction generation and object grounding stable. The authors propose Action QFormer, a small query-based interface sitting between the pretrained backbone and the policy head, which gives action supervision an intermediate route: instruction-conditioned queries reorganize visual evidence into action-facing form before any gradient reaches upstream. In zero-shot sim-to-real navigation, this raises closed-loop task success from 18.8% to 56.3%, raises fixed-instruction action correctness from 22.5% to 75.5%, and nearly eliminates out-of-distribution instruction outputs, while token-level analyses show reduced upstream rewriting with targeted adaptation preserved.

Core claim

On its own terms, the paper's claim is that action supervision should be treated as a representation-shaping force in VLA models, with a dual effect: it is required to make inherited multimodal representations action-compatible, yet when applied directly it also disrupts the same representations' language-side and grounding functions. Action QFormer demonstrates that an instruction-conditioned query interface can mediate this tension by absorbing part of the action-loss update into learnable queries, so that visual information is selected and reorganized for action before gradients propagate back upstream. The empirical demonstration—sim-to-real navigation with a real-scene shift—shows the i

What carries the argument

Action QFormer is a query-based action-facing interface with M learnable query tokens and K transformer layers. Each layer first runs self-attention over queries concatenated with the instruction-side representation, then cross-attention over the image-side representation, so the queries become instruction-conditioned selectors of visual evidence. The updated queries, combined with the instruction representation, form the action-facing representation that conditions a diffusion policy head. The mechanism's work is twofold: forward, it reorganizes inherited multimodal information into action-compatible form; backward, it gives action-loss gradients a query-specific adaptation path, so the ups

Load-bearing premise

The headline numbers come from four hand-picked real-scene scenarios with only eight closed-loop rollouts per condition, scored by manual criteria for success, direction, and collision; if those scenarios or criteria are not representative, the improvement is not established.

What would settle it

Run the same zero-shot comparison on a pre-registered set of, say, 20 scenes with automated success metrics and at least 50 rollouts per condition; if the Action QFormer advantage over direct fusion shrinks to statistical noise, the mechanism-level claim would be refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If action supervision is a representation-shaping force, VLA finetuning pipelines should evaluate not only task success but also how gradients are routed to inherited representations; interface design becomes a control problem over where shaping happens.
  • Action QFormer's gains on directional control and object grounding imply that common VLA failures in new scenes are not purely domain shift—they also reflect poor action-oriented abstraction of visual evidence.
  • The near-elimination of out-of-distribution intermediate instructions suggests that stabilizing upstream representations can substitute for additional real-world instruction data in closed-loop deployment.
  • Mechanistic results imply that the same action objective can either broadly rewrite or locally adapt inherited representations depending on the gradient route, making the polarity of reshaping a design choice rather than a fixed property.
  • Reduced-depth and reduced-query variants still beat the direct-fusion baseline, so even lightweight query interfaces can yield benefit.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the dual-effect mechanism generalizes, the same query-interface principle should transfer to manipulation, where action spaces are higher-dimensional; a natural test is whether Action QFormer-style interfaces reduce upstream rewriting in contact-rich tasks.
  • The attention-stability metric suggests a cheap diagnostic: monitoring reference-region mass during action finetuning could serve as an early warning for disruptive reshaping, independent of task success.
  • The OOD-instruction reduction could be probed further by matching parameter counts and training a non-query adapter; this would isolate whether the benefit comes from the query route itself or merely from extra capacity.
  • The results hint that 'domain shift' in VLA could partly be a representation-organization problem; if true, visual robustness might improve without more data by changing how features are selected, which is testable on standardized benchmarks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes that action supervision in vision-language-action (VLA) models should be understood not merely as a downstream objective but as a force that reshapes representations inherited from pretrained multimodal backbones. It argues that this shaping has a dual effect: it is necessary for forming action-compatible representations, but applied too directly it destabilizes representations that support language-side processing and object grounding. The authors introduce Action QFormer, a query-based action-facing interface with instruction-conditioned learnable queries, inserted between the pretrained backbone and the policy head. They report zero-shot sim-to-real closed-loop navigation improvements (average task success from 18.8% to 56.3%; fixed-instruction action correctness from 22.5% to 75.5%; near-elimination of out-of-distribution instruction generations) and present mechanistic analyses of directional distinction, upstream token rewriting, and instruction-to-visual attention. The central claim is that the query interface changes the route of action-loss gradients, yielding more targeted upstream shaping with less broad rewriting.

Significance. If the empirical and mechanistic claims hold, the paper meaningfully reframes VLA finetuning: instead of treating action supervision purely as a decoder-level objective, it identifies representation shaping as a controllable design axis. The paper has real strengths: a clearly stated mechanism-level hypothesis, a controlled direct-fusion baseline, gradient-blocking diagnostic settings, interface ablations, raw counts in Table V, and multi-level analyses (behavior, token rewriting, attention). These are useful even if the final mechanism is not fully established. However, two load-bearing gaps prevent acceptance in the current form: (1) the baseline and Action QFormer differ in capacity, so the reported gains and reduced rewriting footprint may reflect extra parameters and depth rather than the query-based gradient route; (2) the closed-loop results rest on four hand-selected scenarios with only 8 rollouts per condition and no confidence intervals or inter-rater reliability, and no operational definition of closed-loop task success is given. No code or data is provided, which further limits verification. The paper is honest about several limitations (e.g., the attention-focus effect

major comments (3)
  1. [Sec. III, direct-fusion baseline; Sec. IV-C, ablations; Appendix A.F] The central mechanistic claim—that Action QFormer's query-based interface changes how action-loss gradients shape inherited representations—is underdetermined by a capacity confound. The direct-fusion baseline is a single self-attention pooling operation (Sec. III: SelfAttnFuse(Concat(H_I,H_S)) followed by Pool), whereas Action QFormer inserts M=16 learnable queries and K=4 transformer layers (Appendix A.F). The ablations in Sec. IV-C and Appendix B vary depth, query count, visual source, and instruction combination, but every variant remains inside the query-based family. No direct-fusion adapter with comparable parameter count or depth is tested. Therefore the behavioral gains (Tables II and VI) and the reduced upstream rewriting (Fig. 9, Tables VI–VIII) could be produced by the extra capacity and expressivity of the interface absorbing action-training pressure, rather than by the spec
  2. [Sec. IV-B, Tables II and V; Appendix B] The headline closed-loop results rest on a fragile evaluation. The average task-success improvement from 18.8% to 56.3% corresponds to 6/32 vs. 18/32 rollouts across four hand-selected scenarios. There are no confidence intervals, significance tests, or inter-rater reliability statistics. Moreover, the manuscript does not provide an operational definition of closed-loop 'task success' for the four scenarios; Appendix B defines correctness and collision only for the fixed-instruction action-generation probes, not for the full perception-to-instruction-to-action loop. Given that scoring is manual (Sections IV-B and Appendix B), the 18.8%→56.3% and 22.5%→75.5% claims are not yet established. Please provide scenario-specific success criteria, a blinded or at least consistent scoring protocol with inter-rater agreement where feasible, and confidence intervals or a permutation test on the raw
  3. [Sec. V and Appendix C.A–C.B, action-update-blocked reference] The mechanistic inference in Sec. V relies heavily on comparing a 'full-update' model against an 'action-update-blocked' reference. As stated, the action-update-blocked model is trained without action-loss updates to the inherited pathway, so the comparison between the two reflects different training trajectories, not a clean isolation of the gradient route. Differences in token rewriting and attention could arise from co-adaptation of the policy head, interface, and backbone under different objectives, rather than from the query interface's effect on upstream shaping. The paper acknowledges the reference is 'architecture-matched' only in backbone and policy head, not in adapter capacity. I recommend softening the causal language or adding an intervention that holds total capacity and training dynamics fixed while varying only the gradient route (e.g., stop-gradients on the query path in
minor comments (6)
  1. [Sec. IV-B, Table II] The metrics 'Instruction Direction', 'Action Direction', and 'Instruction OOD Rate' are reported without formal definitions in the main text. Please specify how they are computed (e.g., which model outputs are compared to which references, and what counts as OOD).
  2. [Sec. V-A, Fig. 8] The 'Fisher score' used as a directional-distinction measure is not defined in the text. Please provide the formula and state exactly which representation it is computed on.
  3. [Appendix B.A] The 'quality' metric in the action-generation probes is described only as 'trajectory smoothness'. Please give the formula or scoring rubric, and clarify how it is aggregated across the 8 stochastic samples.
  4. [Figs. 6, 7, 16, 18] Naming is inconsistent across figures: 'Image-latent Action QFormer' / 'Query-only Action QFormer' in the main text correspond to 'QFormer Latent' / 'QFormer Embedding' or 'Action QFormer–Latent' in the appendix. Please unify the terminology.
  5. [Table II vs. Table V] Table II reports 'Average Collisions' while Table V reports 'Number of Collisions'. Clarify that the table values are totals divided by 8 rollouts, and state whether these are raw counts over the entire rollout or per-step collisions.
  6. [Sec. III, overall training objective] The cosine schedule for r_loss_weight is described qualitatively. Please report the actual starting and ending values used in the main runs, and how the schedule interacts with the r=0 and r=1 endpoint cases.

Circularity Check

0 steps flagged

No significant circularity; the main comparisons are controlled experiments and the derivation chain is self-contained.

full rationale

The paper does not fit the circularity patterns. Action QFormer's design is explicitly inspired by BLIP-2/InstructBLIP-style query interfaces (Sec. III, App. A.F), but this is an architectural inspiration, not a load-bearing self-citation, and no uniqueness theorem or prior result by the same authors is invoked to force the choice. The central empirical claims (closed-loop success, fixed-instruction correctness, OOD instruction reduction) are measured against a direct-fusion baseline under matched backbone, policy head, action target, optimizer, and effective batch size; the query interface is not fitted to the reported outcome. The mechanistic analyses compare full-update settings against an action-update-blocked reference within each interface family (Table I, App. C.A.3). This is a controlled gradient-routing comparison rather than a definitional reduction: 'upstream rewriting' is measured as token/dimension drift relative to the reference, and the claim that action-supervised gradients cause that drift is an empirical contrast, not an identity. The only substantive concern is that a capacity-matched direct-fusion adapter is not tested, so the mechanism-level interpretation is underdetermined by an architecture-capacity confound; that is an experimental-design / correctness-risk issue, not circularity. The paper itself appropriately flags qualitative and interpretive claims (e.g., attention-focus examples are 'selectively chosen' and 'interpretive hypothesis rather than a definitive causal claim,' App. C.C.1), further indicating that no central result is being smuggled in as a prediction.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical or theoretical entities are introduced. Action QFormer is a parametric module, and the only conceptual novelty is a framing of action supervision as representation shaping, which is not an entity. The free parameters listed are hand-chosen hyperparameters that the central comparison depends on.

free parameters (3)
  • language-loss weight schedule (r_loss_weight) = Cosine-scheduled from high to low; endpoints not specified in text
    Chosen by hand/training callback; balances instruction generation and action learning and is central to the joint objective (Sec. III; Appendix A-D4).
  • Action QFormer query count M = 16
    Number of learnable query tokens; fixed default without extensive tuning (Appendix A-F); affects capacity to reorganize representations.
  • Action QFormer layers K = 4
    Transformer depth of the query interface; fixed default (Appendix A-F).
axioms (5)
  • domain assumption Hidden states from image and instruction token spans, identified by boundary markers, constitute the complete inherited multimodal representations that matter for action prediction.
    Sec. III: the entire method and diagnostics operate on these extracted spans, ignoring other tokens or global context.
  • domain assumption Comparing a full-update model to an action-update-blocked reference isolates the causal effect of action-supervised representation shaping.
    Sec. V and Appendix C-A3: all rewriting and attention-stability metrics are defined as differences against the architecture-matched action-update-blocked reference.
  • domain assumption GPT-generated instructions, produced from observation plus future trajectory via a manually designed prompt, are valid ground-truth language supervision.
    Appendix A-C: if the prompt biases phrasing or grounding, the OOD-rate and direction metrics inherit that bias.
  • domain assumption A single current-frame observation is sufficient for generating stable instructions and actions.
    Sec. III and Appendix A-B: no temporal context is provided to the model.
  • standard math Diffusion denoising objective, attention backpropagation, and cosine LR scheduling are standard and correctly implemented.
    Sec. III: no formal verification; training relies on standard ML machinery.

pith-pipeline@v1.3.0-alltime-deepseek · 24864 in / 14765 out tokens · 143501 ms · 2026-08-02T01:29:53.248544+00:00 · methodology

0 comments
read the original abstract

Action supervision in vision-language-action (VLA) models is often treated as a downstream objective for learning action prediction. In this paper, we study it instead as a force that shapes inherited multimodal representations. We show that this shaping has a dual effect: it is necessary for forming action-compatible representations, but when action supervision is applied too directly to the inherited multimodal pathway, it can also destabilize representations that support language-side processing and object grounding. To address this tension, we introduce Action QFormer, a query-based action-facing interface that uses instruction-conditioned queries to reorganize inherited multimodal information into action-facing representations before downstream action generation. In zero-shot sim-to-real navigation, Action QFormer improves average closed-loop task success from 18.8% to 56.3%, raises fixed-instruction action-generation correctness from 22.5% to 75.5%, and nearly eliminates out-of-distribution instruction generations. Further analyses show that Action QFormer changes how action supervision shapes inherited multimodal representations, reducing broad upstream rewriting while preserving targeted and sometimes constructive action-supervised adaptation. These results suggest that improving VLA performance requires not only stronger pretrained backbones, but also better ways of selecting and organizing inherited multimodal information while controlling how it is shaped under action supervision.

Figures

Figures reproduced from arXiv: 2607.14635 by Haoyi Niu, Koushil Sreenath, Wenhao Tang, Yi Wu, Yufeng Ji, Zhongyu Li.

Figure 1
Figure 1. Figure 1: Direct versus structured action-loss shaping. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Direct-fusion baseline as a diagnostic reference for action-facing interfaces. (a) The baseline instantiates a common VLA interface pattern: image-side and instruction-side representations inherited from the multimodal backbone are directly fused into an action-facing representation for down￾stream policy prediction. (b) One representative stop-gradient configuration in baseline, where action-loss gradient… view at source ↗
Figure 3
Figure 3. Figure 3: Action QFormer as a query-based action-facing interface. Action QFormer conditions learnable queries on the instruction representation and uses the updated queries to selectively extract action-relevant visual information. The resulting query outputs carry instruction-conditioned visual information and are combined with the instruction-side representation to form action-facing representations for condition… view at source ↗
Figure 4
Figure 4. Figure 4: Four closed-loop real-scene test scenarios (Sec. IV-B). Each panel shows a third-person rollout overlay together with selected first-person observations and generated instructions along the timeline. The four scenarios stress complementary aspects of closed-loop stability: (a) far-end target grounding consistency tests whether a distant target near the edge of the field of view remains grounded despite ear… view at source ↗
Figure 5
Figure 5. Figure 5: Failure progression patterns in representative closed-loop rollouts (Sec. IV-B). Each row shows one real-scene closed-loop rollout of the direct￾fusion baseline, with selected frames arranged from left to right. The left column summarizes the task and dominant failure chain, while the compact tags below each frame preserve the key generated instruction fragment. The two examples illustrate complementary fa… view at source ↗
Figure 6
Figure 6. Figure 6: Fixed-instruction action generation across scenario families (Sec. IV-C). Family-level correctness under fixed-instruction probes on repre￾sentative real-scene observations. Action QFormer improves most strongly on directional control and object grounding, while gains are smaller on motion￾template instructions that require less perceptual disambiguation. viewpoint, rather than simply execute a memorized m… view at source ↗
Figure 7
Figure 7. Figure 7: Representative qualitative comparisons for fixed-instruction action generation (Sec. IV-C). Rows correspond to the three probe families: (a) directional control, (b) object grounding, and (c) motion-template following. In each row, the leftmost image shows the third-person scene context, and the remaining columns show action predictions generated from the same first-person observation and fixed instruction… view at source ↗
Figure 8
Figure 8. Figure 8: The left–right distinction becomes clearest when both visual and language-side pathways are shaped by action-loss gradients (Sec. V-A). We train the direct-fusion baseline while selectively blocking gradients from the action loss to different parts of the multimodal pathway. This plot additionally includes two route-level variants that block gradients to the fused image or instruction representation. (a) L… view at source ↗
Figure 9
Figure 9. Figure 9: Action QFormer reduces upstream token rewriting while preserving targeted adaptation (Sec. V-B). All panels compare the same six gradient settings shown in panel (a). (a) A control/spatial-token view measures token-level cosine distance from the action-update-blocked reference; higher values indicate stronger targeted token modification. The baseline broadly rewrites control-related token groups, whereas A… view at source ↗
Figure 10
Figure 10. Figure 10: Pairwise subspace structure of upstream token rewriting (Sec. V-B). (a) Drift energy measures the overall magnitude of representation rewriting. (b) Effective rank measures how broadly rewriting spans inde￾pendent representation directions. (c) Top-5 variance concentration measures whether rewriting is dominated by a small set of principal directions. (d) Tail@20 variance measures how much drift energy re… view at source ↗
Figure 11
Figure 11. Figure 11: Token rewrite share across token groups (Sec. V-B). We partition tokens into boundary, control, spatial, and other groups, and report the percentage share of total token rewriting assigned to each group. Boundary tokens denote the special tokens inserted at the beginning and end of the instruction span. Compared with the baseline, Action QFormer allocates a larger fraction of rewriting to boundary tokens … view at source ↗
Figure 12
Figure 12. Figure 12: Action QFormer preserves more stable instruction-to-visual attention (Sec. V-C). Each point compares a full-update model with its action-update-blocked reference counterpart. Points in the lower-right region correspond to more stable attention, while the shaded upper-left region marks high-risk attention changes with low reference-region mass and large spatial drift. Compared with the baseline, Action QFo… view at source ↗
Figure 14
Figure 14. Figure 14: Representative qualitative comparisons for interface-capacity ablations under fixed-instruction action generation. Each row shows one testing scenario, with the third-person real scene on the left and action predictions from the same first-person observation and fixed instruction in the remaining columns. Prediction columns compare the Direct-fusion baseline, Reduced-depth Action QFormer, Reduced-query Ac… view at source ↗
Figure 15
Figure 15. Figure 15: Active-dimension structure across gradient-routing comparisons. (a) Token-level active-dimension fraction measures how many hidden dimen￾sions are significantly rewritten per token; lower curves indicate sparser per￾token rewriting. (b) Hidden-dimension active-token fraction measures how many tokens are significantly affected by each hidden dimension; lower curves indicate that most hidden dimensions affe… view at source ↗
Figure 16
Figure 16. Figure 16: Full joint distribution of reference-region mass and spatial drift. The x-axis measures reference-region mass, defined as the fraction of attention from the action-loss-exposed setting that remains inside the action-update-blocked reference’s top-10% attention region. The y-axis measures center shift, defined as the center-of-mass distance between the action-loss-exposed and reference attention maps. The … view at source ↗
Figure 17
Figure 17. Figure 17: Additional attention-alignment scatter statistics. These supporting plots complement the joint analysis in [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Additional attention-alignment CDF statistics. The CDF plots provide distributional views of attention stability under different gradient-routing comparisons. Relative to the direct-fusion baseline, Action QFormer generally exhibits smaller spatial shifts, stronger reference-pattern similarity, higher reference-region mass, lower non-reference-region mass, and stronger overlap with high-response reference… view at source ↗
Figure 19
Figure 19. Figure 19: Additional representative instruction-to-visual attention examples. Each case compares the full-update model with its action-update-blocked reference for the direct-fusion baseline and Action QFormer. (a) The toilet case shows severe baseline attention collapse toward irrelevant visual regions, whereas Action QFormer preserves stronger attention around the target object. (b) The doorway-straight-ahead cas… view at source ↗
Figure 20
Figure 20. Figure 20: Selected per-head instruction-to-visual attention changes for the staircase case in [PITH_FULL_IMAGE:figures/full_fig_p023_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 9 linked inside Pith

  1. [1]

    RT-1: Robotics transformer for real-world control at scale,

    A. Brohanet al., “RT-1: Robotics transformer for real-world control at scale,” inProc. Robot. Sci. Syst., 2023

    2023

  2. [2]

    OpenVLA: An open-source vision-language-action model,

    M. J. Kimet al., “OpenVLA: An open-source vision-language-action model,” inProc. Conf. Robot Learn., 2025

    2025

  3. [3]

    π 0: A vision-language-action flow model for general robot control,

    K. Blacket al., “π 0: A vision-language-action flow model for general robot control,”arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    RT-2: Vision-language-action models transfer web knowledge to robotic control,

    B. Zitkovichet al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,” inProc. Conf. Robot Learn., 2023

    2023

  5. [5]

    π 0.5: A vision-language-action model with open-world generalization,

    K. Blacket al., “π 0.5: A vision-language-action model with open-world generalization,” inProc. Conf. Robot Learn., 2025

    2025

  6. [6]

    FAST: Efficient action tokenization for vision- language-action models,

    K. Pertschet al., “FAST: Efficient action tokenization for vision- language-action models,”arXiv:2501.09747, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    VQ-VLA: Improving vision-language-action models via scaling vector-quantized action tokenizers,

    Y . Wanget al., “VQ-VLA: Improving vision-language-action models via scaling vector-quantized action tokenizers,” inProc. IEEE/CVF Int. Conf. Comput. Vis., 2025

    2025

  8. [8]

    Latent action pretraining from videos,

    S. Yeet al., “Latent action pretraining from videos,” inProc. Int. Conf. Learn. Represent., 2025

    2025

  9. [9]

    Learning to act anywhere with task-centric latent actions,

    Q. Buet al., “Learning to act anywhere with task-centric latent actions,” inProc. Robot. Sci. Syst., 2025

    2025

  10. [10]

    RotVLA: Rotational latent action for vision-language- action models,

    Q. Liet al., “RotVLA: Rotational latent action for vision-language- action models,”arXiv:2605.13403, 2026

    Pith/arXiv arXiv 2026

  11. [11]

    Learning transferable visual models from natural language supervision,

    A. Radfordet al., “Learning transferable visual models from natural language supervision,” inProc. Int. Conf. Mach. Learn., 2021

    2021

  12. [12]

    Align before fuse: Vision and language representation learning with momentum distillation,

    J. Liet al., “Align before fuse: Vision and language representation learning with momentum distillation,” inAdv. Neural Inf. Process. Syst., 2021

    2021

  13. [13]

    BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    ——, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inProc. Int. Conf. Mach. Learn., 2022

    2022

  14. [14]

    Visual instruction tuning,

    H. Liuet al., “Visual instruction tuning,” inAdv. Neural Inf. Process. Syst., 2023

    2023

  15. [15]

    Perceiver: General perception with iterative attention,

    A. Jaegleet al., “Perceiver: General perception with iterative attention,” inProc. Int. Conf. Mach. Learn., 2021

    2021

  16. [16]

    Flamingo: A visual language model for few-shot learning,

    J.-B. Alayracet al., “Flamingo: A visual language model for few-shot learning,” inAdv. Neural Inf. Process. Syst., 2022

    2022

  17. [17]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Liet al., “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” inProc. Int. Conf. Mach. Learn., 2023

    2023

  18. [18]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning,

    W. Daiet al., “InstructBLIP: Towards general-purpose vision-language models with instruction tuning,” inAdv. Neural Inf. Process. Syst., 2023

    2023

  19. [19]

    VEGA: Visual encoder grounding alignment for spatially-aware vision-language-action models,

    H. Wanget al., “VEGA: Visual encoder grounding alignment for spatially-aware vision-language-action models,”arXiv:2605.10485, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    DynaFLIP: Rethinking robotics perception via tri-modal- dynamics guided representation,

    J. Leeet al., “DynaFLIP: Rethinking robotics perception via tri-modal- dynamics guided representation,”arXiv:2605.30350, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    HARP-VLA: Human-robot aligned representation learn- ing for vision-language-action model,

    X. Zhuet al., “HARP-VLA: Human-robot aligned representation learn- ing for vision-language-action model,”arXiv:2605.31234, 2026

    Pith/arXiv arXiv 2026

  22. [22]

    LARA: Latent action representation alignment for vision- language-action models,

    M. Liuet al., “LARA: Latent action representation alignment for vision- language-action models,”arXiv:2606.07100, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Making foresight actionable: Repurposing representation alignment in world action models,

    L. Qiuet al., “Making foresight actionable: Repurposing representation alignment in world action models,”arXiv:2606.12217, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    A formal basis for the heuristic determination of minimum cost paths,

    P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,”IEEE Trans. Syst. Sci. Cybern., 1968

    1968

  25. [25]

    The dynamic window approach to collision avoidance,

    D. Fox, W. Burgard, and S. Thrun, “The dynamic window approach to collision avoidance,”IEEE Robot. Autom. Mag., 1997

    1997

  26. [26]

    ViKiNG: Vision-based kilometer-scale navigation with geographic hints,

    D. Shahet al., “ViKiNG: Vision-based kilometer-scale navigation with geographic hints,” inProc. Robot. Sci. Syst., 2022

    2022

  27. [27]

    GNM: A general navigation model to drive any robot,

    ——, “GNM: A general navigation model to drive any robot,” inProc. IEEE Int. Conf. Robot. Autom., 2023

    2023

  28. [28]

    ViNT: A foundation model for visual navigation,

    ——, “ViNT: A foundation model for visual navigation,” inProc. Conf. Robot Learn., 2023. 13

    2023

  29. [29]

    NoMaD: Goal masked diffusion policies for naviga- tion and exploration,

    A. Sridharet al., “NoMaD: Goal masked diffusion policies for naviga- tion and exploration,”arXiv:2310.07896, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action,

    D. Shahet al., “LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action,” inProc. Conf. Robot Learn., 2022

    2022

  31. [31]

    NaVid: Video-based vlm plans the next step for vision- and-language navigation,

    J. Zhanget al., “NaVid: Video-based vlm plans the next step for vision- and-language navigation,” inProc. Robot. Sci. Syst., 2024

    2024

  32. [32]

    Uni-NaVid: A video-based vision-language-action model for unifying embodied navigation tasks,

    ——, “Uni-NaVid: A video-based vision-language-action model for unifying embodied navigation tasks,” inProc. Robot. Sci. Syst., 2025

    2025

  33. [33]

    Embodied navigation foundation model,

    ——, “Embodied navigation foundation model,”arXiv:2509.12129, 2025

    arXiv 2025

  34. [34]

    Navigation world models,

    A. Baret al., “Navigation world models,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2025

    2025

  35. [35]

    Habitat: A platform for embodied ai research,

    M. Savvaet al., “Habitat: A platform for embodied ai research,” inProc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 9339–9347

    2019

  36. [36]

    ObjectNav revisited: On evaluation of embodied agents navigating to objects,

    D. Batraet al., “ObjectNav revisited: On evaluation of embodied agents navigating to objects,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 6675–6684. 14 SUPPLEMENTARYMATERIALOVERVIEW This supplementary material provides additional implemen- tation, experimental, and mechanistic analysis details. Sec- tion A describes training and impl...

    2020

  37. [37]

    Sequence Construction and Data Flow:Each training sample is organized around asingle current-frame observation, together with its paired language supervision and future action target. The raw language input is first formatted through a chat- style template that encodes the navigation task context, while the image input is processed jointly through the pre...

  38. [38]

    In the main training setup, this parsing uses thespecial-token branchexclusively

    Instruction Parsing and Representation Extraction: After the multimodal backbone produces token-level hidden states, we parse them into the components used by the intermediate interface module before action generation. In the main training setup, this parsing uses thespecial-token branchexclusively. Concretely, the sequence contains explicit boundary mark...

  39. [39]

    Joint Language and Action Supervision:Training com- bines language-side supervision and action-side supervision in a single forward pass. On the language side, the pretrained multimodal backbone is trained under teacher forcing to generate the structured instruction target sequence, with loss computed only on supervised target positions after the prompt p...

  40. [40]

    make a left turn

    Loss Weight Scheduling:In addition to standard optimizer-side scheduling, our training pipeline also supports loss-side schedulingfor balancing language-side and action- side supervision over training. In the main instruction-training setup, the language-side loss weight is dynamically updated during optimization rather than kept fixed. Concretely, the sc...

  41. [41]

    Thedirect-fusion baselineuses the minimal shared-context interface described in Sec

    Interface Families:We compare two interface fami- lies throughout the mechanistic analyses. Thedirect-fusion baselineuses the minimal shared-context interface described in Sec. III, in which inherited image-side and instruction- side representations are fused directly into an action-facing representation. TheAction QFormerfamily replaces direct fusion wit...

  42. [42]

    These set- tings differ in which parts of the inherited multimodal pathway are exposed to action-loss gradients

    Gradient-Routing Settings:Within each interface family, we use four descriptive gradient-routing settings. These set- tings differ in which parts of the inherited multimodal pathway are exposed to action-loss gradients. Action-update-blockedprevents action-loss gradients from updating the inherited multimodal pathway and serves as the reference setting fo...

  43. [43]

    Pairwise Comparison Protocol:Mechanistic quantities such as token-wise rewriting, rewriting-subspace statistics, and attention drift are computed by comparing each action-loss- exposed setting against theaction-update-blockedreference within the same interface family. Each comparison therefore measures how a given gradient-routing setting changes rep- res...

  44. [44]

    Token-wise rewriting and rewriting-subspace analyses examine how strongly the inherited pathway is rewritten and whether that rewriting is broad or targeted

    Use in Mechanistic Analyses:Directional-distinction analysis uses these settings to test whether behaviorally conse- quential distinctions are formed in the action-facing represen- tation. Token-wise rewriting and rewriting-subspace analyses examine how strongly the inherited pathway is rewritten and whether that rewriting is broad or targeted. Attention ...

  45. [45]

    These statistics are supporting measurements: they are not intended to replace the token-level visualizations in Sec

    Token-Level Rewriting Statistics:We first provide com- pact statistics that quantify the upstream token-rewriting trends summarized in the main text. These statistics are supporting measurements: they are not intended to replace the token-level visualizations in Sec. V-B, but instead provide complementary comparisons across gradient-routing settings. Tabl...

  46. [46]

    We partition tokens into boundary, control, spatial, and other groups, and report each group’s share of the total rewriting budget in Table VIII

    Token Rewrite Share Across Token Groups:We further analyze how total token-level rewriting is allocated across token groups. We partition tokens into boundary, control, spatial, and other groups, and report each group’s share of the total rewriting budget in Table VIII. This group-level view complements the focused control- and spatial-token visualiza- ti...

  47. [47]

    We next ask whether this selectivity is also reflected in how rewritten dimensions are organized

    Active-Dimension Structure:The preceding analyses show that Action QFormer reduces broad upstream token rewriting. We next ask whether this selectivity is also reflected in how rewritten dimensions are organized. Rather than exam- ining individual token-level heatmaps, we summarize active- dimension usage from two complementary perspectives: how many hidd...

  48. [48]

    Full Attention-Stability Results Across Gradient-Routing Settings:The main text focuses on the full-update comparison, where attention drift in the direct-fusion baseline is most pronounced. In Fig. 16, we report the full gradient-exposure 20 (a) Token-level active-dimension fraction. (b) Hidden-dimension active-token fraction. Fig. 15.Active-dimension st...

  49. [49]

    These statistics provide additional views of how closely an action-loss-exposed atten- tion map remains aligned with its action-update-blocked refer- ence

    Additional Attention Statistics:To complement the joint reference-region-mass and center-shift analysis, we also report auxiliary attention-alignment statistics. These statistics provide additional views of how closely an action-loss-exposed atten- tion map remains aligned with its action-update-blocked refer- ence. Specifically, we measure global heatmap...

  50. [50]

    18 provides complementary distributional views of attention stability across the full set of phrase-level comparisons

    Distributional Attention Statistics:Fig. 18 provides complementary distributional views of attention stability across the full set of phrase-level comparisons. Unlike the joint scatter plots, which expose relationships between pairs of attention metrics, the cumulative distributions summarize how frequently different levels of alignment and spatial drift ...

  51. [51]

    Additional Attention-Focus Examples:We also include representative attention-focus examples in Fig. 19. These ex- amples provide qualitative illustrations that complement the staircase example in the main text. They cover three regimes: severe baseline attention collapse, baseline attention drift away from the relevant region, and a partially aligned case...

  52. [52]

    20 shows that the all-head average can hide substantial head-level heterogeneity

    Additional Per-Head Attention-Focus Visualization:The per-head visualization in Fig. 20 shows that the all-head average can hide substantial head-level heterogeneity. Some baseline heads exhibit constructive adaptation by concentrating attention near the target object, whereas other heads shift toward irrelevant visual-token regions. For the selected head...