{"id":"d3862fe9-a5aa-4055-948e-2bf89d4f2e38","arxiv_id":"2608.11671","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A vision-language-action policy conditioned on one automatically structured demonstration (sub-goals plus verbalized 3D/2D motion) achieves top scores on LIBERO, LIBERO-Plus, and VLA-Arena without fine-tuning.","lead":"StellaVLA makes a robot control model adapt at test time by giving it one demonstration that has been automatically rewritten into a task plan with sub-goals and verbalized 3D motions. It reports state-of-the-art results on robot benchmarks and a real arm, including adapting from human and virtual-reality demonstrations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Head-to-head comparisons on VLA-Arena and LIBERO-Plus are not matched on test-time demonstration availability: StellaVLA receives a target-task demonstration while the leaderboard baselines are not reported as receiving one, so the claimed SOTA margin may reflect extra test-time information rather…","rationale":"The paper's engineering contribution is real: it provides ablations showing text-only demonstrations nearly match image+text, wrong demonstrations hurt, and cross-source predictions are consistent. These results do not, however, resolve the benchmark asymmetry. The strongest claim in the abstract is a leaderboard rank and specific margins over pi0.5 and LingBot-VLA, and a reader cannot tell whether those margins come from the structured demonstration's reasoning content or simply from giving StellaVLA an extra test-time input that the baselines lack. The no-demonstration ablation that would have resolved this is invalid because of the 0.0 context dropout on LIBERO/LIBERO-Plus; the demonstration-free StarVLA-OFT control suggests the true in-distribution contribution is small, which makes the causal story in Section 3 and the leaderboard claim fragile. This is an externally addressable issue: with matched demonstration availability or a clean no-demo baseline, the conditional can be resolved. I therefore keep the reader's CONDITIONAL verdict. My concern differs from the reader's stated weakest assumption (VLM annotation quality), although the reader also noted the benchmark-matching problem in the rationale; the annotation-quality issue is real but secondary because the text-only/image-only ablations already show the language channel carries the useful signal, and even imperfect annotations could support the observed behavior.","tokens_in":19231,"tokens_out":7462,"duration_ms":82906,"concrete_test":"Re-run the VLA-Arena L0/L1/L2 evaluation for StellaVLA with the demonstration prefix removed, using a checkpoint trained with context dropout 0.5 to match the no-demo deployment distribution, and, if the baseline APIs permit, give pi0.5 the same target-task demonstration (raw or structured) at test time. If StellaVLA's no-demo mean falls toward the 0.44 baseline and/or pi0.5 with a demonstration rises to StellaVLA's level, the leaderboard gap is attributable to test-time demonstration availability rather than to structured reasoning. Separately, train a LIBERO StellaVLA with context dropout 0.5 and repeat the no-demo evaluation to quantify how much of the Table 5 drop is an artifact of train/eval mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that leaderboard-leading scores come from conditioning on one retrieved structured demonstration, so the benchmark comparisons must control for whether a demonstration is supplied at test time. They do not. In Sec. 4.2.2, StellaVLA \"receives one structured demonstration of the target task without any parameter update,\" while the Table 2 baselines are \"from the VLA-Arena leaderboard\" and are not reported as receiving any demonstration. The same asymmetry appears in Sec. 4.2.3 for LIBERO-Plus, where baselines are zero-shot results from [10] but StellaVLA uses a retrieved same-task demonstration. The matched control StarVLA-OFT differs in two variables at once (no demonstration and no spatial-language supervision), so it cannot attribute the gain. The no-demonstration ablation in Table 5 is also confounded: on LIBERO/LIBERO-Plus the context-demonstration dropout is 0.0 (Appendix A), so removing the demonstration at evaluation feeds the model an input it never saw in training; the 62.4% no-demo number is an out-of-distribution input, not a clean control. The demonstration-free StarVLA-OFT reaches 96.6%, suggesting the marginal effect of the demonstration on LIBERO is about 2 points, not the 36 points implied by Table 5. Unless the leaderboard protocol gives every method the same test-time demonstration, the claimed SOTA margin is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StellaVLA, an in-context imitation learning framework in which a raw demonstration is converted offline into a structured demonstration comprising high-level sub-goal descriptions and verbalized 2D/3D kinematic rationales. At test time, one retrieved structured demonstration is prepended to the policy input, with the prefix KV-cached for efficiency. During training, a parallel spatial-language expert supervises the same representation with the sub-task label and a deterministic verbalization of the action chunk; during inference this expert is removed and only the action expert is used. The authors report state-of-the-art results on LIBERO (98.8% average success), VLA-Arena (overall 0.63), LIBERO-Plus (85.1%), and a real-robot benchmark, and argue that in-context structured demonstrations shift imitation from surface-level action mimicry to reasoning-based adaptation.","tokens_in":19574,"tokens_out":3766,"duration_ms":40230,"significance":"If the central claims hold, the paper makes a useful contribution: it offers a practical way to inject structured, language-mediated rationales into VLA policies without retraining at deployment, preserves high-frequency control by avoiding autoregressive decoding, and shows encouraging robustness gains on several benchmarks. The paper is also commendable for including a matched control (StarVLA-OFT), detailed hyperparameters, latency measurements, an ablation of demonstration content, and an explicit limitations section. However, the evidence as presented does not yet establish the headline claims. The main benchmark comparisons do not control for the extra test-time demonstration provided to StellaVLA, the matched control changes two variables at once, and the VLM-generated semantic annotations that are load-bearing for the method are never validated. These are correctness risks rather than presentation issues, so the paper needs substantial revision before the claims can be accepted.","major_comments":[{"comment":"The leaderboard comparisons are not matched on test-time demonstration availability. In Sec. 4.2.2, StellaVLA \"receives one structured demonstration of the target task without any parameter update,\" while the baselines in Table 2 are reported as being from the VLA-Arena leaderboard without any indication that they receive a target-task demonstration. The same asymmetry appears in Sec. 4.2.3, where LIBERO-Plus baselines are zero-shot results from [10] but StellaVLA uses a retrieved same-task demonstration. Since the paper's central claim is that conditioning on one retrieved structured demonstration yields state-of-the-art results, the margin could reflect additional test-time information rather than the method's intrinsic generalization. Please either report leaderboard protocols that give all methods the same demonstration budget, or add a demonstration-free version of StellaVLA under identical conditions for each comparison.","section":"Sec. 4.2.2, Table 2; Sec. 4.2.3, Table 3"},{"comment":"The matched control StarVLA-OFT differs from StellaVLA in two variables at once: it receives neither the retrieved demonstration nor the spatial-language supervision. Consequently, the gains reported against StarVLA-OFT (e.g., LIBERO 96.6 vs. 98.8, LIBERO-Plus 75.0 vs. 85.1, and the real-robot OOD-L1 paired degradation) cannot be attributed to the structured demonstration alone or to the language loss alone. To isolate the contribution of the in-context demonstration, the paper needs a control that adds only the retrieved demonstration to StarVLA-OFT, or a StellaVLA variant trained without the spatial-language loss while still receiving the demonstration.","section":"Sec. 4.2, 'Protocol and Matched Control'; Sec. 4.3"},{"comment":"The correctness of the entire method rests on the offline VLM extraction of sub-goal segmentations and semantic rationale descriptions, but this extraction is never validated. Sec. 3.1 states that Qwen3-VL \"deduces the expert's underlying thought process\" from raw trajectories, yet there is no human agreement study, no quantitative measure of segmentation accuracy, and no failure analysis. The sensitivity demonstrated in Table 5 (a wrong demonstration drops average success from 98.8% to 44.9%) shows how directly the policy depends on the content of the retrieved structured prompt; if the extracted rationales are noisy or systematically wrong, the language supervision in Eq. (2) and the in-context prompt could both be misleading. Please provide an evaluation of annotation quality, e.g., agreement with human segmentations on a sample of trajectories, and an analysis of how annotation errors propagate to downstream success.","section":"Sec. 3.1, Eq. (2), Sec. 4.4, Table 5"},{"comment":"The no-demonstration ablation in Table 5 is not a clean control. According to Appendix A, context-demonstration dropout is 0.0 on LIBERO and LIBERO-Plus, meaning the model was never trained with the demonstration prefix removed. Removing the demonstration at evaluation therefore feeds the model an input distribution it never encountered during training, which explains the large drop from 98.8% to 62.4% without implying that the demonstration contributes 36 points of task specification. This interpretation is supported by the demonstration-free StarVLA-OFT, which reaches 96.6% on the same benchmark and suggests the marginal effect of the demonstration is much smaller. Please report a no-demonstration evaluation using a checkpoint trained with context-demonstration dropout, or explicitly reinterpret Table 5 as an out-of-distribution sensitivity test rather than a causal decomposition.","section":"Table 5 and Appendix A"}],"minor_comments":[{"comment":"The cross-source disagreement values in Table 4 are extremely small, but the table does not report confidence intervals or significance tests. Given that the conclusion is \"source consistency,\" please add error bars or a statistical test over the 1,780 and 1,719 frames.","section":"Sec. 4.3, Table 4"},{"comment":"The terms \"text-only\" and \"image-only\" demonstrations are not defined precisely enough. It should be stated explicitly that the current observation and instruction remain multimodal and that only the demonstration prefix is restricted, so that a reader can judge what information is actually ablated.","section":"Sec. 4.4, Table 7"},{"comment":"The real-robot evaluation uses only 10 rollouts per cell, and the authors themselves note that single cells carry roughly 10-15 points of noise. Please include confidence intervals in Figure 4 or aggregate results with a bootstrap interval, since several cross-method differences appear close to this noise level.","section":"Appendix B"},{"comment":"The claim that the gain on Goal (+3.4) and Long (+3.0) is \"larger\" than on Object (+0.4) is supported, but the table would benefit from confidence intervals or at least the number of rollouts per suite to allow the reader to judge whether the per-suite differences are meaningful.","section":"Sec. 4.2.1, Table 1"},{"comment":"The latency comparison between action-only (88 ms) and language decoding (3177 ms) is presented as a paired measurement, but it is not stated whether the language-decoding condition also uses the cached demonstration prefix. Please clarify the exact setup so the 36x slowdown can be interpreted correctly.","section":"Sec. 3.3, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline results depend on leaderboard and baseline numbers that are not fully controlled, and the core annotation pipeline is not validated. I recommend that the editors ask for (i) a matched-demonstration protocol for the leaderboard comparisons, (ii) an isolation of the demonstration and language-loss contributions, and (iii) a validation study of the VLM-generated rationales. I also note that the paper has a substantial number of self-citations from the same technical team; this is not, by itself, a problem, but it would be good practice to ensure all related work is cited fairly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nStellaVLA is a genuine engineering contribution: the combination of an offline VLM pipeline that turns demonstrations into structured subgoal-plus-verbalized-motion prompts, and a dual-training setup with a parallel language head that is stripped at inference. That is a coherent idea, and the paper executes it broadly - LIBERO, VLA-Arena, LIBERO-Plus, a real-robot benchmark, and several thoughtful ablations (wrong demo, modality, subgoal granularity). The limitations section is unusually candid. Credit where due.\n\nThe soft spots are real and mostly in the benchmarking. The main one: the VLA-Arena and LIBERO-Plus comparisons supply StellaVLA with a target-task demonstration at test time, while the leaderboard/zero-shot baselines are not reported as receiving one. If that is true, the claimed SOTA margin is not a fair comparison, and the margin may largely reflect extra test-time information. The matched control, StarVLA-OFT, changes two variables at once (no demo and no language loss), so it cannot isolate the contribution. And the no-demo ablation in Table 5 is confounded: on LIBERO, context-demonstration dropout is 0.0 during training, so feeding no demo at evaluation is an out-of-distribution input. StarVLA-OFT, which never saw a demo, reaches 96.6% on LIBERO, suggesting the demo's marginal effect is a couple of points, not the 36 implied by Table 5.\n\nThe offline VLM annotation is load-bearing and unvalidated. No human agreement study, no failure analysis of the extracted subgoals. Given the wrong-demo sensitivity (98.8 to 44.9), noisy rationales could mislead the policy.\n\nThe latency claim is also slightly oversold: 'no added latency' in the abstract vs Table 8 showing a cached demo adds ~27ms over no demo. Minor, but fix the wording.\n\nWho is this for: anyone working on in-context imitation or VLA test-time adaptation. The idea of representing demonstrations as structured reasoning is worth taking seriously, and the cross-embodiment consistency data is useful.\n\nRecommendation: send it to peer review. The flaws are addressable with matched baselines and isolated ablations. A serious referee should ask for a version of the leaderboard comparison that controls for demo availability, and a decomposition of the demo and language-loss contributions. After that, the paper could be solid.","headline":"A promising structured-demonstration method for in-context VLA adaptation, but the headline SOTA claims rest on comparisons that give StellaVLA a test-time demonstration the baselines don't get.","tokens_in":20108,"tokens_out":2705,"would_cite":true,"duration_ms":26774,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A robot policy can generalize to new tasks by reading one retrieved demonstration rewritten as the expert's reasoning, the paper reports — first on VLA-Arena, 98.8% on LIBERO, no retraining.","keywords":["vision-language-action models","in-context imitation learning","structured demonstrations","spatial language supervision","robotic manipulation","test-time adaptation without fine-tuning","cross-embodiment transfer","retrieval-augmented policy"],"falsifier":"Take a held-out sample of the same demonstrations, have human annotators segment them into sub-goals, and measure agreement against the automatic annotations; if agreement is low on a substantial fraction of trajectories, the structured-demonstration premise fails. A sharper test that removes the annotation question: feed a trained StellaVLA checkpoint the correct demonstration but with its rationale text randomly permuted across segments — if LIBERO average success stays near the 98.8% baseline, the rationales are not the component carrying the effect.","tokens_in":19039,"feed_emoji":"🤖","tokens_out":18119,"duration_ms":161624,"temperature":0.7,"pith_summary":"This paper claims that a robot policy can adapt out of distribution at test time — no fine-tuning, no weight updates — if it is given one retrieved demonstration rewritten as a structured explanation of the expert's reasoning: a task plan broken into sub-goals, each paired with a verbalized 3D movement and its 2D image-plane trace. The authors argue that prior in-context imitation fails because the policy sees what the expert did without the why behind it, and lacking that structure it falls back on its pretrained habits when the scene, viewpoint, or object changes. StellaVLA conditions the policy on that reasoning, extracted automatically by a vision-language model and reinforced by a training objective that makes the policy state the same rationale while it predicts actions. On this basis the paper reports ranking first on the VLA-Arena leaderboard (0.63 overall versus 0.44 for the strongest baseline), 98.8% average success on LIBERO, 85.1% zero-shot success on the perturbed LIBERO-Plus suite, and real-robot success using human, robot, and XR demonstrations as context. If true, the contribution is a parameter-free route to generalization across both tasks and embodiments: any demonstration source, reduced to the same structured language, can steer a fixed policy.","feed_headline":"One retrieved demo pushes robot policy to top of VLA-Arena","feed_subtitle":"StellaVLA imitates an expert's stated reasoning, not just its motions, and reports 98.8% on LIBERO with no fine-tuning.","key_machinery":"The load-bearing object is the structured demonstration: a rationale-augmented trajectory in which every time step carries its current sub-goal label and its segment's structured rationale. Each rationale has two tiers — a semantic rationale (a sub-goal description such as \"reach for the handle of the blue mug\") and a kinematic rationale produced by the deterministic verbalizer $\\Phi$, which maps any contiguous span of actions to a 3D displacement in the workspace and the 2D projection of that displacement onto the camera plane. Because the same $\\Phi$ is reused on the policy's own predicted action chunks during training, retrieved demonstrations and the policy's predictions share one text vocabulary. Training couples two parallel experts on a single backbone representation $h_t$: an MLP action expert that regresses continuous action chunks under an L1 loss, and the native autoregressive language head that predicts the subtask plus the verbalized movement of the identical chunk under cross-entropy with weight $\\lambda = 0.3$. Since $\\Phi$ is deterministic, both heads supervise the same motion, which is what lets the language objective shape the shared representation without being needed at run time. At inference the language head is removed and the immutable demonstration prefix's key–value attention cache is computed once per episode, so the paper's measured model-side latency stays at 88–91 ms per action chunk instead of the 3177 ms it would cost to decode the language.","core_discovery":"The central claim is that in-context imitation of raw trajectories teaches a VLA policy what an expert did but not why, and that missing why is what blocks out-of-distribution generalization. StellaVLA replaces the raw trajectory with a structured demonstration: an off-the-shelf vision-language model decomposes each demonstration into discrete sub-goals (semantic rationales), and a deterministic verbalizer $\\Phi$ converts each segment's actions into a textual 3D workspace movement and its 2D image-plane trace (kinematic rationales). A parallel dual-training design then sends one shared representation through two heads — a continuous action expert that regresses the next action chunk, and a spatial-language expert that predicts the current sub-goal and the verbalized movement of that same chunk — so the backbone internalizes the reasoning while the language head can be discarded at inference. With one retrieved demonstration supplied as a cached prefix, the paper reports first place on VLA-Arena (0.63 overall versus 0.44 for the strongest baseline), 98.8% average success on LIBERO, and 85.1% zero-shot success on LIBERO-Plus. Ablations support the mechanism: removing the demonstration drops LIBERO average success from 98.8% to 62.4%, and a wrong-task demonstration drops it further to 44.9%, indicating that the policy treats the demonstration as an active task specification rather than inert context.","pith_inferences":["Beyond the paper: the near-flat sub-goal granularity curve (98.1% at three keyframes versus 98.8% at ten) suggests the useful content is the plan skeleton, so the retrieved prefix could probably be compressed to a handful of plan-level text tokens, cutting per-episode memory and latency further.","Beyond the paper: retrieval is purely linguistic, and inside the closed instruction sets of the simulation benchmarks it reduces to exact task matching, so behavior under paraphrased or novel instructions is untested; that regime is the natural next stress test for the framework.","Beyond the paper: the cross-source consistency result covers single-step predictions under matched observations, and the paper itself cautions that small disagreements may accumulate in closed loop; pinning one demonstration source per full rollout would test whether the embodiment-agnostic claim survives the control loop.","Beyond the paper: the dual-training recipe is demonstrated only with an MLP regression action head, so the same language-supervision-forges-the-representation effect remains untested for flow-matching or tokenized action heads; if it transfers, the structured-demonstration idea would port to most VLA families."],"forward_implications":["A fixed checkpoint can be adapted to an unseen task, scene, or object by retrieving one structured demonstration with no parameter updates; the largest LIBERO-Plus gains appear exactly where the paper says OOD failure usually strikes — camera viewpoint (+23.5 points), sensor noise (+19.7), and robot initial state (+14.7).","Demonstrations from other embodiments can steer the policy: under matched observations, switching the demonstration source between real-robot, human-hand, and XR-retargeted versions changes predicted actions by only $0.0014$–$0.0016\\sigma$, which the paper interprets as source-invariant structured context.","The transferable information lives in the structured language, not the demonstration images: text-only demonstrations nearly match the full image+text input at evaluation (98.8/84.4 versus 98.8/85.1 on LIBERO average and LIBERO-Plus), while image-only demonstrations drop to 92.9/75.7.","The demonstration acts as an explicit task specification: supplying the correct demonstration yields 98.8% average LIBERO success, removing it drops to 62.4%, and supplying a wrong-task demonstration drops further to 44.9%, with Goal-suite performance collapsing to 0.0% under the wrong demo.","Structured context does not by itself solve long-horizon error accumulation: on the VLA-Arena Long Horizon suite every method, including StellaVLA, stays near zero at L1 and L2 because a fixed prefix cannot re-plan after execution drifts."],"supporting_citations":[{"why":"Supplies the off-the-shelf vision-language model that automatically segments trajectories into sub-goals and extracts the semantic rationales, the annotation step on which the whole method's premise rests.","marker":"[1]"},{"why":"Defines the LIBERO-Plus seven-axis perturbation benchmark whose published zero-shot baselines are the target StellaVLA must beat to support its out-of-distribution claim.","marker":"[10]"},{"why":"Supplies the fine-tuned OFT-style MLP action-expert recipe that StellaVLA retains as its continuous action head, the basis of its real-time no-language-decoding inference.","marker":"[24]"},{"why":"Defines the LIBERO benchmark whose official evaluation protocol yields the paper's 98.8% in-distribution average success rate.","marker":"[30]"},{"why":"Provides π0.5, the strongest VLA-Arena baseline (0.44 overall) and the external reference model fine-tuned on the same real-robot episodes.","marker":"[37]"},{"why":"StarVLA-OFT is the matched control, trained with the same backbone, action expert, and data as StellaVLA minus the demonstration and spatial-language supervision, isolating the framework's contribution.","marker":"[42]"},{"why":"Provides LingBot-VLA, the second named prior model on the VLA-Arena leaderboard (0.22 overall) that the abstract's headline comparison cites.","marker":"[45]"},{"why":"Supplies the language-action chain-of-thought formulation from which the spatial-language expert's 'subtask plus steering command' supervision objective is retained.","marker":"[52]"},{"why":"Defines the VLA-Arena protocol and leaderboard, the source of the top-line 0.63 overall score and the per-suite task-level generalization numbers.","marker":"[53]"},{"why":"Establishes the prior training-free in-context adaptation approach (Retrieval-VLA) that StellaVLA extends by converting raw trajectories into structured rationales instead of raw observation-action prefixes.","marker":"[57]"}],"fun_headline_variants":["One demo with reasoning pushes VLA to top of leaderboard","Robots learn why, not just how: StellaVLA tops VLA-Arena","Zero-shot OOD adaptation with a single structured demo","In-context reasoning: 98.8% on LIBERO without fine-tuning","Structured demonstrations turn imitation into understanding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the offline vision-language model correctly reading each demonstration — segmenting it into the right sub-goals and describing the right movements — and the paper never measures how often that reading is right; the wrong-demonstration ablation (98.8% down to 44.9%) shows how directly performance depends on prompt quality.","fun_headline_variants_meta":{"raw":{"variants":["One demo with reasoning pushes VLA to top of leaderboard","Robots learn why, not just how: StellaVLA tops VLA-Arena","Zero-shot OOD adaptation with a single structured demo","In-context reasoning: 98.8% on LIBERO without fine-tuning","Structured demonstrations turn imitation into understanding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3380,"prompt_tokens":1177,"completion_tokens":2203,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":793,"completion_tokens_details":{"reasoning_tokens":2129}},"tokens_in":793,"tokens_out":2203,"duration_ms":16466,"temperature":1.0,"reasoning_tokens":2129,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:32:10.708744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out sample of the same demonstrations, have human annotators segment them into sub-goals, and measure agreement against the automatic annotations; if agreement is low on a substantial fraction of trajectories, the structured-demonstration premise fails. A sharper test that removes the annotation question: feed a trained StellaVLA checkpoint the correct demonstration but with its rationale text randomly permuted across segments — if LIBERO average success stays near the 98.8% baseline, the rationales are not the component carrying the effect.","supporting_citations":[{"cited_title":"Retrieval-VLA: Training-Free In-Context Adaptation for Vision-Language-Action Models","cited_arxiv_id":null,"evidence_quote":"Establishes the prior training-free in-context adaptation approach (Retrieval-VLA) that StellaVLA extends by converting raw trajectories into structured rationales instead of raw observation-action prefixes."}],"review_version":1}