REVIEW 3 major objections 5 minor 20 references
FoMoVLA claims that training a VLA policy to predict future visual features and sparse 2D point trajectories, coupled through cross-attention, improves manipulation success and zero-shot robustness while adding no inference overhead.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
FoMoVLA improves VLA robot policies by jointly training future-feature foresight and sparse point tracking, coupled through future-conditioned cross-attention, with auxiliary branches removed at inference.
T0 review reviewed 2026-08-02 challenge →
load-bearing objection Useful VLA training-time auxiliary supervision with a plausible recipe, but the main ablation is confounded by architectural changes (extra tokens, reordered attention), so the headline gains aren't cleanly attributable to the losses. the 3 major comments →
FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that future feature prediction and sparse point tracking are naturally complementary for embodied control: the former says where the scene is heading, the latter says how points in the scene get there, and jointly supervising a VLA backbone with both, tied together by a future-conditioned cross-attention module, produces representations that generate better action chunks. The paper shows that each objective alone improves the base backbone, that combining them yields further gains, and that coupling them through FCCA is what unlocks the largest long-horizon gains. Because the auxiliary branches are training-only, the deployed model is exactly the base backbone plus a sma
What carries the argument
The key machinery is the future-conditioned cross-attention (FCCA) module. FCCA takes the hidden states of K learnable <Foresight> tokens (trained via an EMA teacher to reconstruct the final-frame visual features through a compact MAE decoder) and uses them as keys/values in multi-head attention over the spatial image-token hidden states that feed the point-tracker heads. A zero-initialized output projection makes the module an identity at the start of training, so it injects future-state information into motion prediction gradually. The point-tracking branch itself predicts per-frame 2D displacements and visibility for a sparse 8×8 grid of points, supervised by a frozen point-tracker teache
Load-bearing premise
The frozen point-tracker's 2D trajectories are a reliable, action-relevant description of how task objects move; if those pseudo-labels are systematically wrong, the motion guidance baked into the policy is wrong (and the paper itself notes it only models view-dependent motion in image space).
What would settle it
Train FoMoVLA with the point-tracking supervision replaced by scrambled (randomly permuted or inverted) trajectories from the same teacher, keeping all else equal; if LIBERO-Long and RoboCasa success still rise, the motion signal is not what carries the gain.
If this is right
- If FoMoVLA is right, future-state prediction and motion prediction should be trained together, not as separate auxiliary tasks, because the coupling through FCCA is what produces the largest gains on long-horizon tasks.
- Since all auxiliary branches are discarded at inference, the recipe adds essentially no deployment cost: only 9.4 ms median latency and 0.1 GB memory for the K extra foresight tokens.
- The gains transfer across action-head formulations (flow matching, implicit policy heads), so the supervision recipe is not tied to a specific action decoder.
- The improved tracking accuracy under FCCA (trajectory error roughly halved for moving points) suggests the model learns a more coherent spatio-temporal representation, not just better auxiliary predictions.
- Zero-shot robustness on LIBERO-Plus improves most on language and background perturbations and least on camera and robot-state shifts, consistent with the 2D-imagery nature of the tracking supervision.
Where Pith is reading between the lines
- A natural extension is to replace the 2D point-tracker teacher with 3D trajectory supervision (e.g., from depth or a 3D tracker), which would directly address the paper's stated limitation that camera-viewpoint and robot-state perturbations remain the weakest gains.
- The quantitative tracking metrics reported in the appendix (ATE, Median TE, Survival@10px) could serve as a cheap, rollout-free proxy for tuning foresight quality during training.
- The 'goal state plus motion path' decomposition is a general recipe; it could be ported to navigation, dexterous manipulation, or any policy where the bottleneck is anticipating scene change rather than modeling the actuator.
- The method's reliance on a frozen teacher's pseudo-labels means the ceiling of the approach is bounded by the quality of that teacher; upgrading the teacher should directly raise the policy's ceiling, and using a weaker teacher should cap it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. FoMoVLA augments a VLA policy with two auxiliary training objectives: compact future-feature prediction via K learnable <Foresight> tokens, and sparse 2D point tracking supervised by a frozen CoTracker-v3 teacher, coupled by a future-conditioned cross-attention module. At deployment the auxiliary heads are discarded, but the <Foresight> tokens and a reordered attention mask are retained. The paper reports state-of-the-art LIBERO and RoboCasa GR-1 Tabletop results, plus LIBERO-Plus zero-shot gains, with ablations attributing the improvement to the two auxiliary losses and their coupling.
Significance. The idea of coupling a global future-state representation with sparse per-point motion cues is well motivated, and the method is specified concretely with explicit loss equations, parameter counts, and component-wise ablations. The reported inference overhead is small and the main success metrics are independent of the auxiliary losses, so there is no deep circularity in the headline result. However, the central attribution of the gains to the auxiliary supervision is not yet established because the comparison against the base backbone changes architecture and input sequence, not only losses; statistical support for the small LIBERO margins is also missing. If these issues are resolved, the paper would be a solid contribution to VLA training with predictive and geometric supervision.
major comments (3)
- [Sec. 3.3, 3.5, Supp. A.2, Table 1] The central comparison does not isolate the auxiliary losses. As described in Sec. 3.3 and Supp. A.2, the full model retains K=16 <Foresight> tokens at inference and uses a reordered attention mask (Instruction->Image->Foresight->Action), while the 'Base Backbone' row in Table 1 presumably uses neither. Thus '+ Future Prediction' adds the foresight loss plus 16 extra learnable tokens plus a new attention topology, and '+ Tracking' adds the tracking loss plus the reordering. No ablation controls for the extra tokens or the attention reorder alone. Please run the missing controls, e.g., base + reordered mask without extra tokens, base + 16 randomly initialized learnable tokens with no auxiliary losses, and the full architecture with auxiliary losses disabled. Without these, the paper's claim that training-only supervision causes the gains is not supported.
- [Table 1, Sec. 4.2] Statistical support is missing for the LIBERO claim. With 20 rollouts per task and 10 tasks per suite, per-suite success rates have sizable uncertainty, and many reported differences are small (e.g., Spatial 98.4 vs 97.8; Object 99.6 vs 98.8; the full-model average 98.8 vs 98.5 for several baselines). No confidence intervals, standard errors, significance tests, or multi-seed results are reported for Tables 1-3. The RoboCasa gain (+9.1 pp) is large, but the LIBERO state-of-the-art claim needs explicit error bars or tests to show that the margins are not noise.
- [Supp. B.2, Fig. 6] The tracking evaluation in the appendix is circular. The 'ground truth' for ATE and Survival@10px is the frozen CoTracker-v3 teacher, which is the same teacher used to generate the training targets in Eq. (5)-(7). The reported improvement therefore demonstrates that FCCA makes predicted tracks agree more closely with the teacher's pseudo-labels, not that tracking is more accurate in an independent sense. This does not affect the action-success benchmarks, but the claim that FCCA improves point-tracking accuracy should be reframed as 'agreement with the teacher' or evaluated on independently labeled tracks.
minor comments (5)
- [Sec. 4.3] Typo: 'bythemainbythemain' appears in the paragraph on 'Shared image tokens'. The conclusion also begins with a lowercase fragment ('jointly learning...').
- [Sec. 3.5, Table 8] The text says the auxiliary branches incur 'zero parameter overhead', which is technically true, but inference still processes 16 extra <Foresight> tokens and uses a changed attention mask. The phrase 'training-only' should be qualified as 'training-only losses' to avoid confusion with the retained architectural changes.
- [Sec. 4.4, Table 5] 'Consistently improves' is too strong: for StarVLA-π the FoMoVLA variant decreases on Spatial (-0.6) and Object (-1.0), with the average gain driven by LIBERO-Long (+8.2). Please qualify the scalability claim and discuss the per-suite trade-offs.
- [Sec. 4.5, Fig. 4] The qualitative claim that predicted foresight 'closely reconstructs' ground-truth future features is not backed by a quantitative metric. Reporting the foresight loss or cosine similarity on held-out data would make this claim verifiable.
- [Supp. A.2] The attention-mask design is important for interpreting the results and should be summarized in the main paper, not only in the supplementary material.
Circularity Check
Main benchmark claim is independent, but two auxiliary-task evaluations reduce to the model's own supervision targets (CoTracker teacher as ground truth; EMA self-targets as 'GT' future features).
specific steps
-
fitted input called prediction
[Section 3.2 (Eq. 5) and Supplementary B.2 (Fig. 6)]
"we evaluate models trained with and without FCCA on 10,000 samples from the LIBERO-10 training set using a frozen CoTracker teacher as ground truth."
The same frozen CoTracker-v3 that generates the displacement targets d* and visibility labels v* for the tracking loss Ltrack (Eq. 5) is reused as 'ground truth' in the Fig. 6 quantitative evaluation. Since the model is trained to minimize error to CoTracker's outputs, reporting lower ATE and higher Survival@10px against CoTracker measures how well the model fits its own training labels. This is a fitted-input-called-prediction evaluation: the 'prediction' of point trajectories is scored against the very supervision that produced it, so the reported tracking-quality gains are at least partly by construction and do not independently validate tracking accuracy.
-
self definitional
[Section 3.3 (EMA Teacher) and Section 4.5 / Fig. 4]
"we maintain a shadow copy of the vision encoder updated via exponential moving average (EMA) with momentum µ as the teacher. For the target frame ot+T, the frozen EMA teacher yields target visual features z⋆ ∈ R^{M×d}."
The 'ground-truth' future features z* used in the foresight loss (Eq. 8) are produced by an EMA copy of the model's own vision encoder. Thus 'future feature prediction' is self-distillation: the target is not an independent ground-truth future representation but a slowly moving version of the model itself. When Section 4.5 and Fig. 4 claim the 'predicted foresight closely reconstructs GT future features', the GT is just the EMA teacher output, and the model was explicitly trained to match it. This makes the qualitative 'reconstruction' evidence circular; it cannot by itself establish that the foresight tokens encode task-relevant external future state.
full rationale
The paper's central claim is that FoMoVLA achieves state-of-the-art success rates on LIBERO, RoboCasa GR-1 Tabletop, and LIBERO-Plus. Those results come from external task-success benchmarks and are not logically reduced to the auxiliary losses: the policy is evaluated by environment rollouts, and the auxiliary branches are discarded at inference. Therefore the main derivation is not circular. However, two pieces of evidence used to support the auxiliary representations are circular. First, Fig. 6 evaluates point-tracking quality against the exact frozen CoTracker teacher that generated the training supervision for Ltrack, so the reported ATE/Survival improvements measure fit to the training labels rather than independent tracking performance. Second, the future-feature 'ground truth' in Section 3.3 is generated by an EMA copy of the model's own vision encoder, making the 'predicted vs. GT future features' comparison in Fig. 4 a self-comparison rather than validation against external future observations. These circularities do not invalidate the main benchmark results, but they weaken the paper's internal evidence for the claimed spatio-temporal grounding. Separately, the ablation versus 'Base Backbone' is confounded by the retained 16 <Foresight> tokens and the reordered attention mask (Supplementary A.2), which changes architecture alongside the auxiliary losses; this is a missing-control correctness concern, not a circularity, and is not scored here. Overall, the benchmark-driven central claim remains independently meaningful, so the circularity score is moderate rather than high.
Axiom & Free-Parameter Ledger
free parameters (6)
- lambda1 (foresight loss weight) =
0.1
- lambda2 (tracking loss weight) =
0.3
- lambda_smooth (smoothness weight) =
0.1
- K (number of foresight tokens) =
16
- N (number of tracking grid points) =
64 (8x8 grid)
- EMA momentum mu =
0.999
axioms (7)
- standard math Flow-matching action head with linear interpolation between noise and ground-truth action chunk (Eqs. 1-2) is a valid action decoder.
- domain assumption CoTracker-v3 provides accurate pseudo-labels for point displacement and visibility in robot demonstration videos.
- domain assumption EMA-averaged vision-encoder features of the future frame are a stable, non-degenerate regression target for foresight tokens.
- domain assumption 2D image-space point trajectories are sufficient motion guidance for a 3D manipulation policy.
- domain assumption Reported success rates with 20 (LIBERO) and 50 (RoboCasa) rollouts per task are precise enough to rank methods.
- domain assumption StarVLA-GR00T base backbone and published baseline numbers are directly comparable.
- domain assumption Placing instruction tokens before image tokens in the causal sequence makes image-token hidden states goal-aware.
Cite this review
Pith. "Pith review of FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models." pith.science (2026). https://pith.science/paper/67BUHBZK
@misc{pith2026260714739,
author = {Pith},
title = {Pith review of: FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/67BUHBZK}},
note = {Machine review of arXiv:2607.14739}
}
read the original abstract
Vision-Language-Action (VLA) models have achieved impressive results in visuomotor policy learning, yet remain fundamentally reactive, mapping current observations and language to actions without explicit forward prediction of world dynamics. Existing visual foresight methods predict future visual states but lack explicit motion guidance: they show where to go but not how to get there. We argue that future feature prediction and sparse point tracking are naturally complementary: the former provides the goal state, while the latter captures the continuous motion path toward it. We propose FoMoVLA, a framework that augments VLA representations with explicit spatio-temporal supervision by jointly learning future feature foresight and sparse 2D point tracking, enhancing the continuous action policy. FoMoVLA introduces compact foresight tokens to decode future feature states, decodes sparse temporal 2D point trajectories to model compact geometric motion, and couples both through a lightweight future-conditioned cross-attention module that enables consistent reasoning between anticipated states and point dynamics. Extensive experiments on LIBERO, RoboCasa GR-1 Tabletop, and LIBERO-Plus demonstrate state-of-the-art performance and strong zero-shot generalization. Project page is available at https://liauto-research.github.io/FoMoVLA.
Figures
Reference graph
Works this paper leans on
-
[1]
Bai, S.; Lyu, J.; Zhou, W.; Li, Z.; Wang, D.; Xing, L.; Zhao, X.;Wang,P.;Wang,Z.;Chi,C.;etal.2026.LatentReasoning VLA: Latent Thinking and Prediction for Vision-Language- Action Models. InICML. Bjorck, J.; Castañeda, F.; Cherniadev, N.; Da, X.; Ding, R.; Fan, L.; Fang, Y.; Fox, D.; Hu, F.; Huang, S.; et al
2026
-
[5]
LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. arXiv:2510.13626. Ghosh, D.; Walke, H.; Pertsch, K.; Black, K.; Mees, O.; Dasari, S.; Hejna, J.; Kreiman, T.; Xu, C.; Luo, J.; et al
-
[6]
Point Tracking Improves World Action Models. arXiv:2605.23856. Hu, Y.; Zhang, J.; Luo, Y.; Guo, Y.; Chen, X.; Sun, X.; Feng, K.; Lu, Q.; Chen, S.; Zhang, Y.; Li, W.; and Chen, J
-
[7]
BagelVLA: Enhancing Long-Horizon Manipu- lation via Interleaved Vision-Language-Action Generation. arXiv:2602.09849. Jiang,T.;Jiang,X.;Ma,Y.;Wen,X.;Li,B.;Zhan,K.;Jia,P.; Liu,Y.;Sun,S.;andLang,X.2025. Thebetteryoulearn,the smarteryouprune:Towardsefficientvision-language-action models via differentiable token pruning.arXiv preprint arXiv:2509.12594. Karaev,...
arXiv 2025
-
[9]
Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. arXiv:2601.16163. Kim, M. J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakr- ishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; San- keti, P.; et al
-
[10]
InConference on Robot Learning (CoRL), volume 270, 2679–2713
OpenVLA: An Open-Source Vision- Language-Action Model. InConference on Robot Learning (CoRL), volume 270, 2679–2713. PMLR. Li, F.; Song, W.; Zhao, H.; Wang, J.; Ding, P.; Wang, D.; Zeng, L.; and Li, H. 2026a. Spatial Forcing: Implicit Spa- tial Representation Alignment for Vision-language-action Model. InICLR. Li, L.; Zhang, Q.; Luo, Y.; Yang, S.; Wang, R...
-
[11]
Co- gACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation. arXiv:2411.19650. Lian,S.;Yu,B.;Lin,X.;Yang,L.T.;Shen,Z.;Wu,C.;Miao, Y.;Huang,C.;andChen,K.2026. LangForce:BayesianDe- composition of Vision Language Action Models via Latent Action Queries. InProceedings of the 43rd International Conference...
Pith/arXiv arXiv 2026
-
[12]
HiF-VLA: Hindsight, Insight and Foresight through Mo- tion Representation for Vision-Language-Action Models. arXiv:2512.09928. Liu, B.; Zhu, Y.; Gao, C.; Feng, Y.; Liu, Q.; Zhu, Y.; and Stone, P
-
[13]
Uni- fying Perception and Action: A Hybrid-Modality Pipeline with Implicit Visual Chain-of-Thought for Robotic Action Generation. arXiv:2511.19859. Nasiriany, S.; Maddukuri, A.; Zhang, L.; Parikh, A.; Lo, A.; Joshi, A.; Mandlekar, A.; and Zhu, Y
-
[14]
InRobotics: Science and Systems (RSS)
RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots. InRobotics: Science and Systems (RSS). Physical Intelligence; Ai, B.; Amin, A.; Aniceto, R.; Bal- akrishna, A.; Black, K.; Driess, D.; Finn, C.; Hausman, K.; Levine, S.; et al. 2026.π 0.7: A Steerable General- ist Robotic Foundation Model with Emergent Capabilities. arXiv:2604.15483...
Pith/arXiv arXiv 2026
-
[15]
World Guidance: World Modeling in Condition Space for Action Generation. arXiv:2602.22010. Sun, J.; Zhang, W.; Qi, Z.; Ren, S.; Liu, Z.; Zhu, H.; Sun, G.; Jin, X.; and Chen, Z
-
[16]
VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model. arXiv:2602.10098. Syed, S. N.; Jakobsson, A.; Hao, H.; and Ichnowski, J. 2026.InterceptingtheFuture:Latent-SpacePredictiveWorld Model for Dynamic VLA Manipulation. arXiv:2606.02486. Yang, Y.; Zeng, S.; Lin, T.; Chang, X.; Qi, D.; Xiao, J.; Liu, H.; Chen, R.; Chen, Y.; Huo, D.; et al
arXiv 2026
-
[17]
ABot- M0:VLAFoundationModelforRoboticManipulationwith Action Manifold Learning. arXiv:2602.11236. Ye, J.; et al
-
[18]
StarVLA: A Lego-like Code- base for Vision-Language-Action Model Developing. arXiv:2604.05014. Yuan, H.; Liang, Z.; Chen, A.; Wang, Y.; Li, H.; Lin, P.; Huang, Y.; Lei, Z.; Zhang, T.; Zhang, J.; Zhang, J.; Fan, J.; Zhou, G.; Peng, Q.; Lv, C.; Chen, X.; Yang, A.; Huang, F.; Lin, J.; Liu, D.; Zhou, J.; Wu, C.; and Chen, X.-H. 2026a.Qwen-RobotManipTechnicalR...
-
[19]
DreamVLA: A Vision-Language-Action Model Dreamed with Compre- hensive World Knowledge. arXiv:2507.04447. Zhang, Z.; Luo, W.; Wang, H.; Sheng, Y.; Wang, Y.; Guo, H.; Ren, H.; Du, X.; Che, Y.; Cao, T.; Yuan, L.; and Yu, Y. 2026c. Anticipation-VLA: Solving Long-Horizon Em- bodied Tasks via Anticipation-based Subgoal Generation. arXiv:2605.01772. Zheng, J.; L...
-
[20]
FlowVLA: Visual Chain of Thought-based Motion Reason- ingforVision-Language-ActionModels. arXiv:2508.18269. FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models Supplementary Material A A.1 Model Parameter Breakdown Tab. 6 details the parameter breakdown of FoMoVLA. Dur- ing training, the full model introduces 60.7M add...
-
[2023]
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. InRSS. Fan, J.; Liu, Y.; Li, S.; Ren, B.; Li, S.; Zhang, X.-P.; Ding, W.;andDeng,Z.2026. FUTURE-VLA:ForecastingUnified TrajectoriesUnderReal-timeExecution. arXiv:2602.15882. Fei, S.; Wang, S.; Shi, J.; Dai, Z.; Cai, J.; Qian, P.; Ji, L.; He, X.; Zhang, S.; Fei, Z.; Fu, J.; Gong, J.; and Qiu, X
arXiv 2026
-
[2024]
CoTracker: It is Better to Track Together. arXiv:2307.07635. Kim, M. J.; Gao, Y.; Lin, T.-Y.; Lin, Y.-C.; Ge, Y.; Lam, G.; Liang, P.; Song, S.; Liu, M.-Y.; Finn, C.; and Gu, J
-
[2025]
GR00T N1: An Open Foundation Model for Generalist Hu- manoid Robots. arXiv:2503.14734. Black, K.; Brown, N.; Darpinian, J.; et al. 2025a.π0.5: a Vision-Language-Action Model with Open-World General- ization. arXiv:2504.16054. Black,K.;Brown,N.;Driess,D.;Esmail,A.;Equi,M.;Finn, C.; Fusai, N.; Groom, L.; Hausman, K.; Ichter, B.; et al. 2025b.π 0:AVision-Lan...
Pith/arXiv arXiv 2023
-
[2026]
InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipu- lation. arXiv:2601.02456. Cen,J.;Yu,C.;Yuan,H.;Jiang,Y.;Huang,S.;Guo,J.;Li,X.; Song,Y.;Luo,H.;Wang,F.;etal.2025.WorldVLA:Towards Autoregressive Action World Model. arXiv:2506.21539. Chi, C.; Xu, Z.; Feng, S.; Cousineau, E.; Du, Y.; Burch- fiel, B.; Tedrake, R.; and Song, S
arXiv 2025
This paper was first reviewed by deepseek-v4-flash on August 2, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.