{"id":"a93966ea-f78c-462f-a796-2ec65ba5f13c","arxiv_id":"2505.11214","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OE-VLA extends vision-language-action models to follow open-ended instructions embedded in images, videos, and goal snapshots, matching text-only performance on the CALVIN benchmark.","lead":"A new robot control model, OE-VLA, lets a robot follow instructions given as images, videos, or handwritten notes instead of only typed text. The authors build two test suites and show the model handles these open-ended instructions almost as well as text-only commands.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported open-ended success rates are measured on benchmarks built by the same automatic transformation pipeline and from the same CALVIN distribution used for training, so the central claim of general open-ended multimodal instruction understanding is not yet established.","rationale":"The paper's central contribution is a model that consumes multimodal instructions, and that claim is only as strong as the evaluation's ability to distinguish genuine instruction understanding from in-distribution pattern matching. The same-pipeline benchmark issue directly threatens this distinction: the training data and the OE-CALVIN benchmarks are generated by the same automatic transformations over the same CALVIN environment, so high scores could reflect memorization of pipeline-specific cues rather than generalization to open-ended human instructions. Other weaknesses, such as missing code and weights, absence of error bars, lack of comparison with VIMA and Interleave-VLA, and the 7B model's regression on VGR in OE-CALVINhard, are important for reproducibility and completeness but do not undercut the central claim as directly. If the proposed human-annotated evaluation showed robust transfer, the core claim would stand despite those gaps; if it failed, the headline numbers would need to be reinterpreted as measuring a narrower, template-bound skill. Since the reader's weakest_assumption already identifies this same concern and the conditional verdict already reflects it, no verdict change is needed.","tokens_in":15183,"tokens_out":7748,"duration_ms":78148,"concrete_test":"Obtain or retrain the OE-VLA7b model using the released recipe, then evaluate it on a held-out set of multimodal instructions for the same CALVIN D tasks where the instructions are created by independent human annotators rather than by the Section 4.2 pipeline: annotators photograph objects in a new scene, write commands by hand, and record goal images and demo videos from a different viewpoint. If the average successful sequence length on this independently constructed set falls by more than roughly 15% relative to the reported OE-CALVINbase value of 3.48, the open-ended result is substantially an artifact of the shared pipeline and distribution. If performance remains within 15%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that OE-VLA genuinely understands open-ended multimodal instructions, rather than exploiting a narrow data recipe. The evidence for this rests on OE-CALVINbase and OE-CALVINhard, but both benchmarks are produced by the same transformation pipeline described in Section 4.2 and from the same CALVIN data distribution used in Stage-2 training. In OE-CALVINbase, object images are crops from raw environmental observations, optical instructions use plain backgrounds and regular fonts, and goal images and videos come from the same environment and perspective (Section 5.1). The training data for VOS, OIF, VGR, and VDL are built with the same detection, rendering, and frame-sampling code. Consequently, the model can succeed by exploiting low-level correlations with the training distribution: for example, matching the pixel appearance of a cropped object against the current observation, recognizing the exact rendering style of the OCR text, or using temporal frame identity in the video demo. The OE-CALVINhard benchmark does not resolve this because it only varies surface attributes (web images, handwritten fonts, new viewpoints) while still applying the same automatic construction recipe. Without an evaluation on instructions generated independently of the authors' pipeline, the reported 3.48 and 2.68 average sequence lengths cannot be attributed to general open-ended instruction understanding. This is load-bearing because the paper's core novelty claim is precisely the ability to handle free-form multimodal human instructions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OE-VLA, a vision-language-action model built on LLaVA-Next-Interleave that accepts four categories of open-ended multimodal instructions in addition to language: visual object specification (VOS), optical instruction following (OIF), visual goal reaching (VGR), and video demo learning (VDL). The method fine-tunes the foundation model in two stages: a multi-image grounding stage on MGrounding and an open-ended instruction-tuning stage on data automatically transformed from CALVIN with the Section 4.2 pipeline. The authors also introduce two new benchmarks, OE-CALVINbase and OE-CALVINhard, generated from CALVIN by replacing language annotations with multimodal instructions. Experiments report an average successful sequence length of 2.99 on CALVIN ABC to D with text input for OE-VLA7b, 3.48 on OE-CALVINbase, and 2.68 on OE-CALVINhard.","tokens_in":15312,"tokens_out":8007,"duration_ms":66493,"significance":"If the claims were supported, the paper would make a useful contribution: a single VLA that can follow image, optical-text, goal-image, and video-demo instructions would broaden the interaction modes for robot manipulation. The data-construction recipe is simple, general, and potentially reusable, and the introduced benchmarks would provide a common evaluation suite. The paper also avoids a separate policy head, which eases reproducibility. However, the empirical evidence is currently insufficient because both benchmarks are generated by the same automatic pipeline used to produce the training data, no baselines that accept multimodal prompts are evaluated, and results are reported without variance. The central claim of open-ended instruction understanding therefore rests on an in-distribution evaluation; the paper's value as a benchmark contribution will depend on addressing this gap in revision.","major_comments":[{"comment":"The two new benchmarks are produced by the same transformation pipeline and from the same CALVIN data distribution used for Stage-2 training. Section 4.2 describes object crops from detections on the raw data, rendered text via Python plotting libraries with sampled fonts and positions, uniform frame sampling for video demos, and 80-frame goal segments. Section 5.1 then defines OE-CALVINbase with object crops from environmental observations, optical instructions with plain backgrounds and regular fonts, and goal images and videos from the same environment and perspective; OE-CALVINhard only varies surface attributes (web images, handwritten fonts, new viewpoints) while keeping the same construction recipe. The success rates in Tables 2 and 3 thus reflect the model's ability to follow instructions generated by the authors' own code, not its ability to handle independently generated open-ended instructions. Since the abstract and Section 1 claim open-ended multimodal instruction understanding, the evaluation set should be independent of the training pipeline, for example human-elicited instructions or a second generator with different object detection, rendering, and sampling code. Without such an evaluation, the 3.48 and 2.68 average sequence lengths cannot be attributed to general open-ended understanding.","section":"§5.1 vs §4.2"},{"comment":"No existing method that accepts multimodal prompts is used as a baseline. Section 2 identifies VIMA and concurrent Interleave-VLA as closest in motivation, but neither is evaluated on OE-CALVINbase or OE-CALVINhard. The paper's claim that no previous VLA models have been capable of handling such open-ended tasks would be better supported by at least one comparative experiment, or by a baseline that decomposes the problem, such as OCR/LLM transcription of optical instructions feeding a language-conditioned policy, or a simple object-image matching policy for VOS. Without such comparisons, the reader cannot determine whether the OE-VLA architecture itself is necessary for the reported performance.","section":"§5.2.2, Tables 2-3"},{"comment":"The discussion compares the OE-CALVINbase average length (2.75 for OE-VLA1b) with the language-conditioned result on CALVIN (2.70), stating that the multimodal result is even slightly higher than the language-conditioned baseline. This mixes the OE-CALVINbase and CALVIN benchmark distributions: even if the underlying physical tasks are the same, the instruction modality and the instruction-generation process differ, so the comparison is not controlled. A valid comparison would evaluate the same model on the same task sequences with language and multimodal prompts under an identical protocol, or would explicitly frame the numbers as cross-benchmark and avoid claiming that multimodal instructions are handled at least as well as language.","section":"§5.2.2"},{"comment":"All reported results are single runs with no error bars, multiple seeds, or significance tests. Differences such as OE-VLA7b's 2.99 versus KosMos Interleave's 2.70 on CALVIN, or the stage-1 ablation improvements in Figures 5 and 6, could be within stochastic training variance. The paper should report the mean and standard deviation over at least three seeds for the main comparisons, and ideally a paired setting for ablations, before claiming the best performance or quantifying the benefit of the two-stage pipeline.","section":"§5.2.1, Tables 1-4 and Figures 5-6"},{"comment":"The manuscript repeatedly refers to an appendix ('Further details are provided in the Appendix'; 'Additional details regarding these benchmarks and other experiments are provided in the Appendix'), but no appendix is present in the submitted manuscript. Since the paper's contributions include a data-construction recipe and two benchmarks, the missing appendix blocks reproduction and verification of the construction pipeline, including the exact detection model, rendering parameters, frame sampling choices, hyperparameters, and benchmark construction code. This needs to be supplied before the reproducibility claims are supportable.","section":"§4.3, §5.1"}],"minor_comments":[{"comment":"The foundation model name is rendered inconsistently as 'LLaVA-Next-Interleave', 'LLaV A-Next-Interleave', and 'LLaVA-Interleave-next'; please standardize the notation.","section":"§4.1"},{"comment":"Both 'SigLip' and 'SigLIP' are used; please unify the spelling.","section":"§4.1, Eqs. (1)-(2)"},{"comment":"The text says a 0.5B-parameter model was trained, but the tables and discussion use 'OE-VLA1b'; please clarify whether the smaller model is 0.5B or 1B and keep the naming consistent.","section":"§5.2"},{"comment":"The captions describe 'upper row' and 'lower row', but the figures are arranged as panels (a)-(d) with two bar charts per panel; please re-describe the layout to avoid ambiguity.","section":"Figures 5-6"},{"comment":"The statement 'we trained all our models for one epoch for a fair comparison' should specify what is being compared and why one epoch is fair, since different models may require different numbers of epochs to converge.","section":"§5.2.4"},{"comment":"Equation (5) defines the action likelihood over n actions, but the text mentions a five-step action chunk; please specify that n=5.","section":"§4.1, Eq. (5)"}],"recommendation":"major_revision","confidential_remarks":"The contribution is useful in principle, but the evaluation is currently in-distribution with respect to the authors' own data-generation pipeline, and the missing appendix and baselines further weaken the reproducibility and comparative value. I would encourage the authors to add an externally generated evaluation set, one or two strong multimodal-prompt baselines, multiple seeds with error bars, and the missing appendix; the paper could then be a solid benchmark contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhao et al. build a single VLA that takes object images, rendered texts, goal frames, and short video demos as instructions alongside text, and they show it still does well on CALVIN's language-only benchmark. The genuinely new part is the automatic recipe that rewrites a language-labeled dataset into these four instruction types, and the two OE-CALVIN suites that go with it. That recipe is simple and likely to be reused by others; the two-stage fine-tuning on MGrounding before robot data is a reasonable choice, and the ablations show it helps. Text-conditioned results are competitive: 2.99 average sequence length on ABC->D beats the no-diffusion-head baselines they compare against. So the paper earns its place as a practical systems contribution.\n\nThe soft spots are in what the paper claims beyond that. 'Open-ended' is doing a lot of work: the instructions are templated prompts with image placeholders, and the OE-CALVINbase instructions are generated by the exact same object-crop, OCR-render, frame-sampling code applied to the same CALVIN distribution as the training data. So base results mostly show the model has learned the pipeline's formatting, not that it understands arbitrary human multimodal instructions. OE-CALVINhard is better, since it uses web images, handwritten fonts, and new viewpoints, but it still comes from the authors' own recipe. The paper also has no error bars or seeds, no comparison to VIMA (which takes multimodal prompts, though object-centric) or to the concurrent Interleave-VLA, and no code or benchmark release, which makes the central numbers hard to trust or verify. The 7B model's drop on VGR-hard relative to the 1B (1.25 vs 1.29) is a small crack in the scaling story, but it's one task.\n\nI don't think any of this is fatal. The core result—that a VLA can be fine-tuned to follow several visual instruction formats with acceptable performance—is credible and useful. But 'general open-ended multimodal understanding' is not established by this evaluation. The paper is for robot learning researchers who want a starting point and a data pipeline for multimodal prompting; those readers will get value from it. It deserves a serious referee, mainly to force comparisons and artifact release, and to tighten the claims.","headline":"A practical data-recipe contribution that shows a VLA can handle several visual instruction formats, but the 'open-ended' claim is stronger than the evidence because the benchmarks are built by the same transformation pipeline as the training data.","tokens_in":16000,"tokens_out":3900,"would_cite":true,"duration_ms":35876,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single vision-language-action model can follow instructions given as images, whiteboard text, goal pictures, and video clips, not just typed language.","keywords":["vision-language-action model","multimodal instructions","robot manipulation","open-ended instruction following","CALVIN benchmark","visual goal reaching","video demonstration learning","curriculum learning"],"falsifier":"Run OE-VLA7b on a held-out set of instructions written by humans who never saw the training recipe, including object photos from unseen scenes, handwritten notes, goal images from different robots, and videos of tasks absent from CALVIN. If the average successful sequence length drops below the language-only baseline of 2.99, or near the 1.25 score seen for hard visual goal reaching, then the open-ended gains are artifacts of the synthetic instruction distribution rather than evidence of general multimodal instruction understanding.","tokens_in":2081,"feed_emoji":"🤖","tokens_out":4218,"duration_ms":109103,"temperature":0.7,"pith_summary":"This paper tries to establish that one vision-language-action policy can obey instructions that are not written as sentences: a photo of an object to grasp, a command rendered as text inside an image, a goal picture of the desired state, and a short video demonstration. The proposed model, OE-VLA, is fine-tuned from a multimodal language model that accepts interleaved images and text, with actions discretized into language-like tokens so no separate policy head is needed. On the CALVIN benchmark with ordinary text instructions, the 7-billion-parameter version completes an average of 2.99 consecutive subtasks, matching or exceeding strong language-conditioned baselines; on the authors' new OE-CALVINbase benchmark it reaches 3.48, and on the harder OE-CALVINhard benchmark 2.68. The paper also contributes two benchmarks and a general recipe for converting language-annotated manipulation datasets into multimodal-instruction data.","feed_headline":"One robot policy takes orders as images, notes, and video demos","feed_subtitle":"It keeps text-only performance at 2.99 average subtasks and reaches 3.48 on its own multimodal benchmark.","key_machinery":"The central mechanism is a unified token sequence: the robot observation, the text parts of the instruction, and every instruction image are each encoded and concatenated in their original order, and the action chunk is discretized into 256 bins drawn from rare language tokens so the whole problem becomes next-token prediction. This lets one architecture handle four instruction modalities without object detectors, separate action heads, or task-specific modules. The supporting machinery is the data recipe: a pipeline that turns any language-annotated robot dataset into multimodal instruction data, plus a two-stage fine-tuning schedule that starts from a multi-image grounding dataset before training on robot data.","core_discovery":"OE-VLA is built on a large multimodal model that ingests a robot observation, with static and wrist camera views concatenated into one image, together with a prompt that can interleave text and any number of images. All inputs are tokenized into one sequence, and robot actions are discretized into 256 bins that reuse rare language tokens, so the model predicts an action chunk autoregressively. Four open-ended instruction types are supported: visual object specification, optical instruction following, visual goal reaching, and video demo learning. To train this, the authors convert CALVIN's language annotations into these forms with an automatic pipeline using object crops, rendered text with varied fonts and backgrounds, goal frames, and uniformly sampled video frames, then fine-tune in two stages. The paper reports that with text-only input OE-VLA7b reaches an average successful sequence length of 2.99 on CALVIN ABC to D, the best among the compared models without a separate action head, while with open-ended instructions it averages 3.48 on OE-CALVINbase and 2.68 on OE-CALVINhard, with visual goal reaching consistently the weakest category.","pith_inferences":["Editorial inference: because OE-CALVINbase and OE-CALVINhard are generated from the same CALVIN scenes and the same transformation code used to build training data, the open-ended scores are best read as measuring how well the model follows this specific instruction recipe; a set of instructions written by independent human annotators would be a stricter test of true open-endedness.","Editorial inference: the fact that OE-VLA7b's visual goal reaching collapses on the hard benchmark to an average length of 1.25 while other modalities stay near 3 suggests the model may rely on low-level appearance matching for goal images; a test that changes object colors, lighting, or camera viewpoint while keeping the semantic goal identical would separate genuine goal understanding from perce","Editorial inference: a direct extension would apply the same transformation pipeline to a larger real-robot manipulation dataset and measure whether open-ended multimodal instruction following transfers across embodiments, since the paper demonstrates the recipe only on CALVIN.","Editorial inference: the text-only fine-tuned models in Table 4 outperform the mixed-modal models on language, so the recipe's roughly equal per-modality sampling is not yet balanced; a better mixing ratio or a curriculum over modalities could raise both language and open-ended performance."],"forward_implications":["Because the same architecture handles text, object images, rendered commands, goal images, and demo videos, human-robot interaction can use whatever modality is most natural in the moment, such as pointing at an object, drawing an arrow, or showing a phone video, without adding a new model per modality.","Language ability does not have to be traded away: with text-only input OE-VLA7b reaches 2.99 average successful sequence length on CALVIN ABC to D, and a text-only fine-tuned version reaches 3.49, so conventional language control remains alongside the new instruction forms.","The two new benchmarks, OE-CALVINbase and OE-CALVINhard, give later work a fixed target for open-ended instructions, with the hard version using web-sourced object images, handwritten-style commands, and out-of-environment goal images and videos.","The data-recipe claim implies that other language-annotated manipulation datasets can be converted to multimodal-instruction form, so the approach may scale to real-robot data beyond CALVIN.","Scaling helps open-ended instructions more than language-only instruction: OE-VLA7b's open-ended average of 3.48 exceeds its already strong text average of 2.99 on the base benchmark, whereas visual goal reaching remains the bottleneck in both model sizes."],"supporting_citations":[{"why":"Supplies the LLaVA-Next-Interleave foundation model that accepts interleaved multi-image and text input, the base OE-VLA is fine-tuned from.","marker":"[Li et al., 2024c]"},{"why":"Provides the CALVIN benchmark and evaluation protocol whose ABC to D split and chain-success metric the paper uses for all experiments.","marker":"[Mees et al., 2022]"},{"why":"Establishes the VLA paradigm of fine-tuning a vision-language model to output discretized action tokens, which OE-VLA follows.","marker":"[Zitkovich et al., 2023]"},{"why":"Provides the MGrounding multi-image grounding dataset used in stage-1 training to improve spatial perception.","marker":"[Li et al., 2025]"},{"why":"OpenVLA serves as a primary language-conditioned baseline that OE-VLA compares against and outperforms in the ABC to D setting.","marker":"[Kim et al., 2024]"},{"why":"VIMA is the closest prior multimodal-prompt robot policy, and the paper positions OE-VLA against it by listing the architectural differences.","marker":"[Jiang et al., 2023]"},{"why":"Supplies the SigLIP vision encoder used to tokenize both observations and instruction images.","marker":"[Zhai et al., 2023]"},{"why":"Supplies the Qwen-1.5 LLM backbone and tokenizer, whose vocabulary the action-token scheme reuses.","marker":"[Bai et al., 2023]"}],"fun_headline_variants":["Robot follows instructions from images, text, and demos","Beyond text: robot acts on multimodal prompts","OE-VLA: robot policy for open-ended multimodal commands","Robot handles image, note, and video instructions","Multimodal instructions unlock robot's open-ended control"],"cache_read_input_tokens":17920,"weakest_assumption_plain":"The load-bearing assumption is that the open-ended evaluation benchmarks measure real instruction following, when in fact they are generated by the same automatic transformation pipeline, object crops, rendered text, goal frames, and video frames, applied to the same CALVIN environment used for training.","fun_headline_variants_meta":{"raw":{"variants":["Robot follows instructions from images, text, and demos","Beyond text: robot acts on multimodal prompts","OE-VLA: robot policy for open-ended multimodal commands","Robot handles image, note, and video instructions","Multimodal instructions unlock robot's open-ended control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1364,"prompt_tokens":980,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":309}},"tokens_in":596,"tokens_out":384,"duration_ms":3879,"temperature":1.0,"reasoning_tokens":309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:55:26.910028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OE-VLA7b on a held-out set of instructions written by humans who never saw the training recipe, including object photos from unseen scenes, handwritten notes, goal images from different robots, and videos of tasks absent from CALVIN. If the average successful sequence length drops below the language-only baseline of 2.99, or near the 1.25 score seen for hard visual goal reaching, then the open-ended gains are artifacts of the synthetic instruction distribution rather than evidence of general multimodal instruction understanding.","supporting_citations":[{"cited_title":"Sanketi, Grecia Salazar, Michael S","cited_arxiv_id":null,"evidence_quote":"Establishes the VLA paradigm of fine-tuning a vision-language model to output discretized action tokens, which OE-VLA follows."},{"cited_title":"VIMA : Robot Manipulation with Multimodal Prompts","cited_arxiv_id":null,"evidence_quote":"VIMA is the closest prior multimodal-prompt robot policy, and the paper positions OE-VLA against it by listing the architectural differences."}],"review_version":1}