{"id":"20a7d9e9-c5cb-4194-957f-0470d2ec4ece","arxiv_id":"2608.08839","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"SG-WAM injects text-grounded and spatial-aware semantic foresight from a VLM planner into a world-action model, improving instruction-following in simulated and real robot manipulation.","lead":"This paper adds a vision-language model planner to robot world-action models, producing semantic scene guidance so robots follow language instructions in both predicted video and action control. The authors report leading scores on LIBERO and LIBERO-Plus and on a dual-arm robot for tasks like stacking and ordered picking.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never tests whether the predicted foresight actually anticipates the future: SigLIP2/DA3 features are static-dominated, so current-frame features may suffice and the central 'future foresight' mechanism is unestablished.","rationale":"I take the paper's mechanism claim at face value: a VLM planner converts the current observation and instruction into future dense features that condition a world-action model. The architecture is coherent, and the three-stage training avoids teacher-forcing at inference by conditioning on the planner's own predictions. However, the load-bearing premise is untested. SigLIP2 and Depth Anything 3 features are heavily influenced by static scene content; in LIBERO, the relevant objects are visible in the current frame, so the planner can minimize Eq. (3) without genuinely predicting state change. The ablations remove guidance entirely but do not isolate future anticipation from current-frame grounding. This is exactly the reader's weakest assumption, so I agree with the reader. The missing direct comparison with S-VAM and the absence of repeated-seed statistics are additional reasons to remain conditional, but the feature-shortcut issue is more fundamental because it concerns whether the proposed mechanism actually does what the paper claims. A single diagnostic comparing predicted versus current and future teacher features would settle the question: if predicted features are closer to future features, the concern is resolved; otherwise the central claim needs revision. The reader's CONDITIONAL verdict is therefore appropriate, and my read does not change it.","tokens_in":12081,"tokens_out":8212,"duration_ms":88323,"concrete_test":"Using the Stage-1-trained planner on held-out LIBERO episodes, compute the mean cosine similarity and L2 distance between the predicted Fhat_tg/Fhat_spa and (a) teacher features from the current frame, and (b) teacher features from each of the K future keyframes. Then rerun full SG-WAM inference with the predicted foresight replaced by the current-frame teacher features through the same projection and injection. If the predicted features are not significantly closer to the future features than to the current features, or if current-frame guidance reproduces the reported ~98.7% LIBERO success, the future-foresight mechanism is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 trains the planner to reproduce, from the current frame and instruction, dense features that frozen SigLIP2 and Depth Anything 3 extract from actual future keyframes (Eq. 3). The central claim is that this constitutes 'semantic foresight' that steers video and action generation to follow the instruction. The paper provides no evidence that the predicted features are closer to future teacher features than to current-frame teacher features. In tabletop manipulation, the target object and container are usually visible in the current frame; SigLIP2 and DA3 features of future keyframes are dominated by the same static appearance and layout, so the planner can score a low alignment loss by emitting near-current features with modest instruction grounding. The Table 3 ablation removes guidance entirely and loses 1.5 points on LIBERO; it does not test whether the future component, rather than injection of current-frame semantic and geometric features, causes the gain. No per-keyframe error versus temporal offset is reported, so it is unknown whether the foresight degrades with horizon. Without a diagnostic comparing predicted features with current versus future teacher features, or an ablation replacing predicted foresight with current-frame features, the statement that actions and videos 'faithfully follow' the instruction because of future anticipation is unsupported. If the concern lands, the method may still work as an observation-grounding mechanism, but the headline claim of semantic foresight would be inaccurate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SG-WAM augments world-action models (WAMs) with semantic foresight predicted by a VLM planner. From the current observation and language instruction, the planner outputs two dense feature maps—text-grounded foresight aligned to SigLIP2 features of future keyframes and spatial-aware foresight aligned to Depth Anything 3 features—using learnable query tokens (Eqs. 2–3). These maps are fused via a learned gate and injected into the video expert through cross-attention, while the action expert inherits the guidance through joint attention. Training proceeds in three stages: planner pretraining, planner+video co-training, and full co-training with the action expert. The authors evaluate on LIBERO (98.7% average, Table 1), LIBERO-Plus (81.3% average, Table 2), and a dual-arm real-world setup, with ablations in Table 3 and qualitative instruction-following studies on DROID. The central claim is that text-grounded and spatial-aware foresight makes both future-video generation and action prediction follow the language instruction.","tokens_in":12341,"tokens_out":5725,"duration_ms":55131,"significance":"If the mechanism is as claimed, SG-WAM is a plausible and well-motivated alternative to observation-independent text embeddings in WAMs. The use of frozen teacher encoders on actual future frames as supervision is methodologically sound, and the three-stage training is a sensible way to preserve pretrained generative ability. The LIBERO-Plus robustness gains and the real-world qualitative results are encouraging. However, the paper's central contribution is not yet established: the evidence does not show that the predicted foresight is truly anticipatory rather than a repackaging of current-frame features, and the LIBERO SOTA claim rests on a 0.2-point margin without error bars. The quantitative ablations are also small and unreplicated. If the additional diagnostics requested are provided, the work could be a solid contribution; as it stands, the significance is conditional.","major_comments":[{"comment":"The SOTA claim on LIBERO rests on a 0.2-point margin (98.7 vs. 98.5) over LingBot-VA, and no standard errors, confidence intervals, or number of seeds are reported for any table in the paper. Given typical run-to-run variance in policy training, this margin may be within noise. Please report mean ± std over at least three seeds for SG-WAM and the strongest baselines, or equivalently provide confidence intervals, and adjust the claim accordingly.","section":"Section 4.2 (Table 1)"},{"comment":"The paper does not provide evidence that the planner's predictions are actually anticipatory. The training target in Eq. (3) is features extracted from future keyframes, but no comparison is made between the predicted features and current-frame teacher features. In tabletop manipulation the scene is largely static, so SigLIP2 and Depth Anything 3 features of future keyframes may be dominated by static appearance, and the planner could achieve low alignment loss by emitting near-current features with modest instruction grounding. The 'w/o semantic guidance' ablation removes guidance entirely (97.2 vs. 98.7), so it does not isolate the future component. Please add (a) a similarity comparison of predicted features to current vs. future teacher features as a function of keyframe offset, (b) an inference-time ablation that replaces the predicted foresight with current-frame teacher features, and (c) if feasible, per-offset prediction error to demonstrate degradation with horizon. Without such diagnostics, the claim that future anticipation, rather than observation grounding, drives the instruction-following gains is unsupported.","section":"Section 3.2 (Eq. (3)), Section 4.5 (Table 3)"},{"comment":"The claim that the text-grounded and spatial-aware branches provide complementary information is not substantiated by the ablations: removing either branch changes success rate by only 0.4-1.0 points, and no statistical significance is reported. Moreover, the learned gate g in Eq. (4) is not reported; if the sigmoid output is near 0 or 1, the two branches may not be balanced as intended. Please report the learned value of g and quantify the agreement/correlation between the two predicted maps, so that the reader can assess whether the two semantics are actually distinct.","section":"Section 3.2 (Eq. (3)) and Section 3.3 (Eq. (4))"}],"minor_comments":[{"comment":"The implementation states that the planner predicts K=4 future keyframes at evenly spaced offsets, but the offsets are not specified; please report the actual frame offsets (or time horizon) used.","section":"Section 4.1"},{"comment":"The planner is described as applied to each camera view independently with shared weights, but the manuscript does not explain how the foresight maps from multiple camera views are combined in the world model; please clarify the fusion mechanism.","section":"Section 3.2"},{"comment":"The guidance tokens receive a positional encoding of keyframe time and spatial location; please specify how this encoding is constructed (e.g., sinusoidal or learned).","section":"Section 3.3"},{"comment":"The single-stage training ablation is described only as 'single-stage training'; please clarify whether this means all components are trained jointly from random initialization or with some warm-start.","section":"Section 4.5"},{"comment":"Real-world success rates are reported without error bars or confidence intervals; since there are only 50 trials per setting, reporting a binomial confidence interval would aid interpretation.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The reader's concern about circularity is unfounded: the planner is supervised by frozen teacher features from actual future frames, not the model's own outputs, and the teachers are discarded at inference. However, the missing diagnostic on whether predicted features are truly anticipatory is real and load-bearing. The paper also relies on a very thin SOTA margin without error bars. The requested additions—feature-comparison diagnostics, current-frame-feature ablations, and seed-or-CI reporting—are within the scope of a major revision and would substantially strengthen the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SG-WAM proposes a VLM planner that, from the current frame and instruction, predicts dense features from frozen SigLIP2 and Depth Anything 3 for future keyframes, then injects those features into a world-action model via gated cross-attention. That is a concrete, well-described design, and the three-stage training is sensible. The ablations show the guidance matters (about 1.5 points on LIBERO), and the real-robot results are directionally consistent. It deserves a serious referee.\n\nWhere it falls short is in the strength of its claims. The headline LIBERO number is 98.7% vs 98.5% for LingBot-VA, with no error bars, repeated seeds, or confidence intervals. That margin is within typical run-to-run noise. Calling it SOTA on that basis is not supportable. The 'semantic foresight' framing is also under-tested. The planner is trained to reproduce future-frame features, but there is no diagnostic showing the predictions are actually closer to future features than to current-frame features. SigLIP2 and DA3 features in tabletop scenes are dominated by static layout, so the method may be working as observation grounding rather than true anticipation. Replacing the future target with a current-frame target in an ablation would settle this; the paper does not do it. The instruction-following analysis is qualitative and thin. Also, the paper cites S-VAM from the same group but never discusses it in related work, despite S-VAM proposing self-distilled geometric and semantic foresight. That is a positioning gap.\n\nFor all that, the architecture is coherent and the training objective is well-defined. No circular reasoning is apparent. I would send this to peer review, but with the expectation of heavy revision: release code and data, report seed-level statistics, add the current-vs-future diagnostic, and directly compare with S-VAM. As is, the quantitative claims outrun the evidence, but the core idea is worth engaging.","headline":"A clean, well-specified method for injecting VLM-predicted semantic features into a world-action model, but the 'future foresight' claim is underevidenced and the headline LIBERO margin is too thin to call SOTA without error bars.","tokens_in":602,"tokens_out":1286,"would_cite":true,"duration_ms":33247,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A VLM planner's text-grounded and spatial-aware semantic foresight steers world-action models to follow instructions, reaching 98.7% average success on LIBERO.","keywords":["world-action models","semantic foresight","vision-language model planner","instruction grounding","spatial-aware guidance","robot manipulation","LIBERO benchmark","feature alignment"],"falsifier":"An experiment that would settle the claim: build a LIBERO variant where the target object named in the instruction is occluded at time t0 and only becomes visible in the future keyframes; if SG-WAM's success does not drop toward the no-guidance baseline, the planner is not actually anticipating the future from the scene, and the gains come from something else (e.g., the instruction alone or static appearance cues).","tokens_in":11863,"feed_emoji":"🤖","tokens_out":5548,"duration_ms":52624,"temperature":0.7,"pith_summary":"World-action models generate future video and actions jointly, but most condition on text embeddings computed independently of the scene, so the generated behavior drifts from the instruction. SG-WAM proposes inserting a vision-language model as a semantic planner that, from the current frame and instruction alone, predicts dense feature maps of what the future should look like: one map aligned with language-grounded image features and one with geometric depth features. These maps are injected into a pretrained video diffusion world model through cross-attention, and the action decoder inherits them through joint attention, so both the imagined video and the executed actions follow the instruction. The paper reports state-of-the-art results on LIBERO (98.7% average success), the highest robustness on LIBERO-Plus perturbed tasks (81.3%), and successful real-robot transfer, with the largest gains when instructed objects must be selected among distractors or handled in a commanded order.","feed_headline":"VLM foresight steers world-action models to 98.7% on LIBERO","feed_subtitle":"A semantic planner predicts text-grounded and spatial-aware future features that guide video and action generation.","key_machinery":"The load-bearing mechanism is a query-token VLM planner. Base query tokens plus two modality-specific groups predict K=4 future keyframes at evenly spaced offsets in a single forward pass; a resampler cross-attends to query states and image states to produce 256 tokens in each of two feature spaces. The text-grounded target comes from penultimate-layer patch tokens of SigLIP2 (a language-aligned dense visual encoder), and the spatial-aware target from dense features of Depth Anything 3 (a dense geometry encoder), with both detached and used as teacher targets under an MSE/smooth-L1 loss. The two predicted maps are projected and gated into a single guidance, injected into every video-expert block via parallel cross-attention while retaining original text conditioning, and the action expert receives guidance through joint attention with the video expert. Three-stage training first pre-trains the planner, then co-trains it with the video expert, then adds the action expert, always conditioning on the planner's own predictions, which matches deployment.","core_discovery":"The central claim is that a VLM planner can translate a language instruction and the current observation into a semantic foresight—text-grounded and spatial-aware feature maps of future keyframes—and that injecting this foresight into a world-action model makes both future-video generation and action prediction faithfully follow the instruction. On LIBERO the full model reaches 98.7% average success, edging out LingBot-VA (98.5%); on LIBERO-Plus it tops all baselines with 81.3% average under seven perturbation types; and on a real dual-arm platform it beats GE-Act and FastWAM on all four tasks plus two generalization settings. Ablations show the largest drop when semantic guidance is removed entirely (97.2%), and smaller drops when either the text-grounded or spatial-aware branch is removed.","pith_inferences":["A testable extension: since the planner predicts future features from the current frame alone, the same architecture could be trained to predict foresight at multiple horizons and then select the horizon whose downstream action success is highest, effectively doing closed-loop temporal abstraction.","The teacher-feature design raises a question the paper does not settle: whether SigLIP2 and Depth Anything 3 features are the minimal sufficient supervision, or whether a single encoder with both properties (e.g., a VLM with dense metric heads) would yield the same gains with fewer tokens.","If the gains come mainly from grounding object identity, a lighter substitute—such as an open-vocabulary object detector with geometric priors—might capture much of the benefit at lower VLM cost; the 0.4-point gap between the full model and the no-spatial variant (98.7 vs 98.3) suggests spatial features add a smaller but real increment.","The largest real-world gains on ordered, multi-object tasks suggest the method's value concentrates where instructions specify relationships and order; applying it to tasks with sequential language subgoals might compound the benefit."],"forward_implications":["If SG-WAM is right, world-action models can be steered by language without retraining the video backbone from scratch: a pretrained video diffusion model can be conditioned by injected semantic foresight while keeping its generative priors.","The single-forward-pass planner means no test-time future sampling or iterative refinement is needed to obtain guidance, keeping inference cost close to a standard WAM.","Because the action expert needs no direct guidance injection, the same semantic foresight could be reused across different action decoders attached to the same video backbone.","Instruction grounding in feature space rather than in pixel or text space should transfer across appearance shifts, as suggested by the LIBERO-Plus language and lighting perturbations and the real-world height and lighting generalization results.","The method's success under language rephrasing (81.7% on that LIBERO-Plus perturbation) suggests that grounding the instruction in the current scene, rather than embedding it independently, is what drives robustness to rephrasing."],"supporting_citations":[{"why":"Supplies SigLIP2, the frozen teacher whose penultimate-layer patch tokens define the text-grounded foresight target.","marker":"Tschannen et al. (2025)"},{"why":"Supplies Depth Anything 3, the frozen teacher whose dense features define the spatial-aware foresight target.","marker":"Lin et al. (2025)"},{"why":"The Qwen3.5 VLM backbone that the semantic planner is instantiated on.","marker":"Qwen Team (2026)"},{"why":"LTX-Video, the pretrained video diffusion model that initializes the world model backbone.","marker":"HaCohen et al. (2025)"},{"why":"LIBERO benchmark supplying the four evaluation suites and the standard protocol for success-rate reporting.","marker":"Liu et al. (2023)"},{"why":"LIBERO-Plus benchmark supplying the seven perturbation factors used for the robustness evaluation.","marker":"Fei et al. (2025)"},{"why":"Genie Envisioner, whose joint-attention coupling of an action decoder to a video model is the basis for the action expert design.","marker":"Liao et al. (2025)"},{"why":"DROID dataset used to train the RGB-based WAM and SG-WAM for the out-of-distribution instruction-following comparison.","marker":"Khazatsky et al. (2024)"},{"why":"FastWAM, a WAM baseline used for real-robot comparisons and for the joint video-action architecture reference.","marker":"Yuan et al. (2026)"}],"fun_headline_variants":["VLM planner injects foresight for text-grounded robotic actions","Semantic foresight from VLM boosts world-action models to 98.7%","Spatial-aware foresight lets robots follow language, not just pixels","Text-grounded foresight aligns videos and actions to instructions","VLM foresight gives world-action models 98.7% on LIBERO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method stands on the premise that the feature maps of future keyframes extracted by frozen SigLIP2 and Depth Anything 3 are the right and sufficient supervision: a single VLM forward pass over the current frame and instruction can anticipate exactly those features, and they contain the instruction-relevant object identity and geometry needed for the task.","fun_headline_variants_meta":{"raw":{"variants":["VLM planner injects foresight for text-grounded robotic actions","Semantic foresight from VLM boosts world-action models to 98.7%","Spatial-aware foresight lets robots follow language, not just pixels","Text-grounded foresight aligns videos and actions to instructions","VLM foresight gives world-action models 98.7% on LIBERO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000634,"raw_usage":{"total_tokens":2921,"prompt_tokens":938,"completion_tokens":1983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":1886}},"tokens_in":554,"tokens_out":1983,"duration_ms":15801,"temperature":1.0,"reasoning_tokens":1886,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:22:33.815180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that would settle the claim: build a LIBERO variant where the target object named in the instruction is occluded at time t0 and only becomes visible in the future keyframes; if SG-WAM's success does not drop toward the no-guidance baseline, the planner is not actually anticipating the future from the scene, and the gains come from something else (e.g., the instruction alone or static appearance cues).","supporting_citations":[],"review_version":1}