{"id":"0117397b-8787-4d9b-a83e-7ca4734d1352","arxiv_id":"2505.00693","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Hand-drawn symbolic sketches (arrows and circles) can serve as precise, silent robot instructions, and a vision-language pipeline can execute them on unseen tasks.","lead":"This paper introduces RoVI, a hand-drawn sketch language of arrows, circles, and colors that can command robots silently and with precise spatial meaning. The authors also present VIEW, a pipeline that turns these sketches into robot actions, and report high success rates on unseen manipulation tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The promised 87.5% execution rate rests on an unexplained dissociation between VLM language comprehension (38%) and code execution; without per-trial evidence that correct code accompanies wrong language, the central claim is not supported.","rationale":"The reader's weakest assumption identifies exactly the same soft spot: the execution success cannot be explained if the VLM's semantic understanding is as poor as Table 2 reports. I agree with the conditional verdict because the paper's central contribution is the RoVI-to-action pipeline, and its headline number depends on a dissociation that is asserted rather than demonstrated. The concern is not an external disagreement with consensus; it is an internal inconsistency in the reported results and a missing link in the causal chain from RoVI to action. A per-trial analysis or a ycode ablation would settle this directly. I do not recommend changing the reader's verdict because the requested revisions (code/data release and a mechanism for the gap) are already part of the conditional acceptance criteria; no additional adjustment is needed.","tokens_in":14749,"tokens_out":3031,"duration_ms":34819,"concrete_test":"Release per-trial logs for the LLaVA-13B (RoVI Book) real-world evaluation: for each of the 10 trials per task, record ylanguage, ycode, extracted keypoints, and execution outcome. Then compute the conditional execution success rate restricted to trials where task/planning comprehension was scored incorrect. If this rate is not near 87.5%, or if no trial pairs an incorrect ylanguage with a ycode that actually implements the ground-truth action, the claimed dissociation fails. Additionally, rerun Task 6-8 with ycode replaced by a fixed keypoint-following primitive while keeping keypoints identical; if success is unchanged, VLM code generation is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim hinges on VIEW-LLaVA-13B succeeding at 87.5% real-world execution while scoring only 38% on task/planning comprehension (Table 2 vs Table 1). Section 6.3 asserts that the execution function maps action and sequence errors, making it unaffected by perception errors, but no mechanism or per-trial evidence is provided. This matters because Eq. (2) defines ylanguage and ycode as outputs of the same fθ, conditioned on the same RoVI and observation. If the VLM misidentifies the task or planning in 62% of trials, there is no evident reason its generated Python code encodes the correct action. The paper offers no examples of wrong-language/correct-code trials, no code-correctness metric, and no ablation varying ycode while holding keypoints fixed. Without such evidence, the high execution success could be explained by the keypoint module and low-level policy doing the real work, with the VLM's code output largely irrelevant or only superficially checked. If the code is irrelevant, the strong claim about VLMs understanding RoVI is overstated; if the code is relevant, the comprehension results contradict the execution results. Either way, the current evaluation does not establish the claimed dissociation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Robotic Visual Instruction (RoVI), a hand-drawn symbolic language for robot task specification that uses arrows, circles, colors, and numbers to encode spatial-temporal constraints. The authors also present VIEW, a pipeline that combines a VLM interpreting RoVI into language and Python code, a YOLOv8-based keypoint extractor, and a low-level keypoint-conditioned controller. They contribute a 15K-instance dataset (RoVI Book) for fine-tuning small VLMs, and report experiments across 11 tasks in real and simulated environments, claiming up to 87.5% success on unseen real-world tasks and outperforming language-conditioned baselines.","tokens_in":15168,"tokens_out":5255,"duration_ms":48436,"significance":"The paper addresses a real limitation of natural-language instruction for human-robot interaction—spatial ambiguity, verbosity, and inappropriateness in quiet settings—and proposes a concrete visual alternative with a user-friendly sketch interface. If the reported results are reliable, the work would be a useful step toward sketch-based HRI and provides a valuable open dataset as well as a plausible edge-deployment recipe via LoRA fine-tuning. The evaluation includes multi-step tasks, disturbances, and trajectory-following requirements, which are relevant stress cases. However, the central quantitative claim rests on a poorly evidenced dissociation between language comprehension and code execution, and the empirical evaluation has statistical and baseline-selection weaknesses. These issues are addressable but currently leave the headline claim under-supported.","major_comments":[{"comment":"The paper reports that VIEW-LLaVA-13B (RoVI Book) achieves 38% average task/planning comprehension (Table 2) yet 87.5% real-world execution success (Table 1). Since Eq. (2) defines ylanguage and ycode as outputs of the same fθ on the same input, low language comprehension should be expected to correlate with errors in the generated Python code. The explanation in Section 6.3 that \"the execution function maps action and sequence errors, making it unaffected by perception errors\" is not supported by any mechanism or per-trial evidence. The manuscript provides no examples of trials with incorrect language but correct code, no code-correctness metric, and no ablation that holds keypoints fixed while varying ycode. This is load-bearing because the 87.5% success rate is the paper's headline claim; without such evidence, it is unclear whether the VLM's code output contributes to execution or whether the keypoint module and low-level policy alone drive the results. Please provide per-trial annotations of language, code, and execution outcomes, a code-correctness evaluation, or an ablation that replaces or corrupts ycode.","section":"Section 6.3, Table 2 vs Table 1, Eq. (2)"},{"comment":"All success rates are computed over 10 trials per task, with no error bars, confidence intervals, or significance tests. Several close comparisons (e.g., Task 4: VIEW-GPT4o 60% vs CoPa 60%; Task 7: VIEW-GPT4o 90% vs VIEW-LLaVA-13B 90%) cannot support the claim of \"significant generalization capability\" at this sample size. Please report per-trial data, confidence intervals, or statistical tests (e.g., exact binomial tests) for the headline comparisons.","section":"Section 6.1, Table 1"},{"comment":"The evaluation omits existing sketch-based and visual-prompting methods such as RT-Sketch [45], MOKA [36], and ReKEP [28], despite citing them in Related Work. Since the paper's claim is that RoVI is a better visual instruction paradigm than natural language and goal images, the absence of a direct comparison against prior visual/sketch-conditioned policies makes the comparative conclusion incomplete. Please add at least one such baseline, or justify why the selected baselines suffice.","section":"Section 6.1, Section 6.2, Related Work"},{"comment":"The table footnote states that VIEW∗ denotes both VIEW-GPT4o and VIEW-LLaVA-13B \"as their test results are identical.\" It is surprising that two different VLMs produce exactly identical results across all three simulator tasks. If the simulator tasks are essentially solved by the keypoint module and code templates independent of the VLM's comprehension, that supports the concern in the first major comment rather than the paper's interpretation. Please clarify how identical results arise and what this implies for the role of the VLM in VIEW.","section":"Section 6.2, Table 1 (Simulator)"}],"minor_comments":[{"comment":"The name \"LLaV A\" appears with a spurious space in several places; it should be \"LLaVA.\"","section":"Throughout"},{"comment":"The row labeled \"Small Models\" has all-zero scores but does not identify which models are included; this row should either be removed or broken down by model.","section":"Table 2"},{"comment":"Figure 7 is referenced in the text but no detailed description is given, and the \"detailed statistics\" are deferred to supplementary material that is not provided with the manuscript; please include the relevant statistics in the main text or an appendix.","section":"Figure 7"},{"comment":"The section mentions in-context learning but provides no details or examples of the in-context prompts; these should be included in the appendix so the evaluation is reproducible.","section":"Section 6.3"},{"comment":"The notation is inconsistent: v is defined as an image in R^{H×W×3} in Section 5.1 but Section 5.3 writes \"v∈ R2 in pixel space\"; please correct the variable or the dimension.","section":"Section 5.3"},{"comment":"The free parameter ε (keypoint reach threshold) is never specified; please report its value and, ideally, its sensitivity in the experiments.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim may be recoverable with additional per-trial analysis of the language/code/execution relationship. I do not see grounds for rejection if the authors provide the missing evidence. Also, the manuscript repeatedly refers to a supplementary document that is not included; the authors should be asked to provide it. Given the paper's stated contributions, a direct comparison with RT-Sketch and related visual-prompting methods would substantially strengthen the comparative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The RoVI representation is worth knowing about, and the 87.5% real-world number is more honest than most, but the paper doesn't explain why a VLM that fails task comprehension two-thirds of the time still executes well.\n\nWhat's new is the combination: object-centric arrows and circles with color-coded temporal steps, plus keypoint extraction feeding a code-generated low-level policy. That is a concrete advance over RT-Sketch and MOKA, which use sketches or marks but don't tie them to temporal sequencing or code-conditioned keypoint control. The 15K RoVI Book dataset and the ablations on drawing style and keypoint detection are genuinely useful and could seed follow-up work.\n\nThe main soft spot is Section 6.3. LLaVA-13B scores 38% on task/planning comprehension but 87.5% on execution. The paper asserts that the execution function maps action and sequence errors, making it unaffected by perception errors, but provides no per-trial examples or a code-correctness metric. Since y_language and y_code come from the same f_theta, a skeptical reader needs to see at least one trial with wrong language and correct code, plus a measure of code correctness. Without that, the gap could mean the code output is a shortcut that bypasses real understanding, which would undercut the claim that VLMs can interpret RoVI.\n\nThe evaluation is also thinner than it should be: 10 trials per task with no error bars, and the baselines are language-conditioned only. There is no comparison to RT-Sketch or MOKA, which are the closest prior art. That is a gap in both the experiments and the related-work discussion. One minor point: GPT-4o generated the training data and is also used for evaluation, creating some circularity, but the real-world results are measured against external baselines, so I don't think that is load-bearing.\n\nThis paper is for people working on visual instruction, human-robot interaction, or code-conditioned manipulation. The RoVI idea could be a useful primitive, but the evaluation needs strengthening before the central dissociation claim is accepted. I'd send it to peer review rather than desk reject it, because the idea is serious and the ablations are indicative. A major revision should add per-trial analysis, the missing baselines, and error bars.","headline":"A useful new interaction primitive with a real evaluation gap: the execution-vs-comprehension dissociation needs per-trial evidence.","tokens_in":15516,"tokens_out":2503,"would_cite":true,"duration_ms":25379,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a hand-drawn sketch language of arrows, circles, colors, and numbers can replace natural language for commanding robots with pixel-level spatial precision.","keywords":["robotic visual instruction","human-robot interaction","vision-language models","keypoint-conditioned policy","hand-drawn sketches","robot manipulation","spatiotemporal constraints","parameter-efficient fine-tuning"],"falsifier":"Run the pipeline on a fixed set of RoVI sketches, then deliberately corrupt the model's natural-language task and planning responses while keeping the generated code and keypoint coordinates unchanged; if execution success drops correspondingly, the claim that execution is unaffected by perception errors is refuted, and if it stays high, keypoints and code are carrying the task.","tokens_in":14603,"feed_emoji":"✏️","tokens_out":7981,"duration_ms":76618,"temperature":0.7,"pith_summary":"The paper proposes replacing natural-language robot commands with a hand-drawn visual language, RoVI, where arrows give trajectories, circles mark affordances, colors order steps, and numbers label arms. It then builds VIEW, a pipeline that turns such sketches into executable actions: a vision-language model names the task and writes Python function calls, a keypoint detector pulls pixel coordinates from the arrows and circles, and a keypoint-conditioned low-level policy moves the arm through those coordinates. To make small models practical, the authors collect a 15K-instance instruction dataset and fine-tune a 13-billion-parameter vision-language model. On eleven unseen tasks in simulation and the real world, the method reports up to 87.5% real-world execution success, outperforming language-conditioned baselines in cluttered scenes, multi-step operations, and trajectory following. The paper is establishing that a sketch, not speech, can be a precise and silent command channel for everyday robot manipulation.","feed_headline":"Hand-drawn sketch commands get robots to 87.5% on unseen tasks","feed_subtitle":"Circles, arrows, and color codes give pixel-level spatial cues that speech blurs, beating language prompts in cluttered scenes.","key_machinery":"The load-bearing mechanism is the RoVI symbol set combined with the keypoint-conditioned policy. An arrow is formally an ordered list of pixel coordinates, starting point, intermediate waypoints, and endpoint, while a circle contributes a single affordance center; color numbers the temporal steps. The keypoint module detects these symbol parts directly in the image, so the spatial constraints come from the drawing rather than from object detection, and the pixel coordinates are lifted to 3D with depth data and converted into end-effector poses in SE(3). A cost function with a translation term and a rotation term then moves the arm from one keypoint to the next, while the vision-language model's generated Python functions decide which action each keypoint sequence represents.","core_discovery":"The paper's central claim is that a hand-drawn 2D sketch can serve as a complete robot instruction: an arrow encodes a trajectory through its tail, waypoints, and head; a circle marks where to grasp, rotate, or press; and color assigns each symbol to a step in a multi-step task. VIEW is the pipeline that reads that sketch: a vision-language model produces a task name, a step-by-step plan, and Python function calls; a keypoint module extracts the arrow and circle coordinates from the image; and a low-level policy lifts those pixel coordinates to 3D and drives the end effector through them by minimizing a translation-and-rotation cost. The reported discovery is that this combination transfers to tasks and environments the system has not seen, reaching 87.5% real-world success on trials that demand multi-step sequencing, tolerance of clutter and disturbances, and exact trajectory following. The paper further claims that execution success does not depend on the vision-language model's language comprehension, because the code path is separate from the keypoint grounding.","pith_inferences":["A stricter test of the paper's separation claim would corrupt only the natural-language output while leaving keypoints and code intact; if success collapses, the language path is not as independent as claimed.","The same visual grammar could be extended with new primitives, for example a dashed line for 'wait here' or a cross for 'avoid', suggesting RoVI may be a general-purpose sketch language rather than a task-specific trick.","A user study that times sketching versus typing instructions and measures error rates would show whether the spatial-precision advantage survives use by non-experts.","Pairing RoVI with a short verbal goal, such as a sketch of where plus a word for what to do, could combine the convenience of speech with the precision of drawing; the paper leaves this hybrid unexplored."],"forward_implications":["Users can specify exact grasp points, paths, and end states by drawing on a live camera image, which removes the ambiguity of phrases like 'near' or 'below'.","Long-horizon tasks become tractable because color-coded steps let the pipeline split one complex instruction into single-step subtasks.","Quiet environments such as libraries and hospitals become viable settings for robot instruction, since no speech is required.","Edge deployment becomes plausible because a 13-billion-parameter model fine-tuned on the 15K dataset can drive the whole pipeline.","Because the keypoint module tracks symbols rather than objects, performance degrades less in cluttered scenes with distractors than language-conditioned approaches do."],"supporting_citations":[{"why":"Supplies the high-capacity vision-language model used both to generate dataset answers and as the main interpreter in real- and simulated-task experiments.","marker":"[1]"},{"why":"Provides the base manipulation episodes and images from which the 15K-instance RoVI Book training set is created.","marker":"[13]"},{"why":"Provides the 13-billion-parameter vision-language model that is fine-tuned on RoVI Book and reports the highest real-world execution success.","marker":"[37]"},{"why":"Serves as a language-conditioned baseline whose composable value-map planning is compared against VIEW on spatial-precision tasks.","marker":"[27]"},{"why":"Serves as another language-conditioned baseline using object-part spatial constraints and visual marking, used in the real-world comparison.","marker":"[26]"},{"why":"Supplies an end-to-end vision-language-action model used as a language-conditioned baseline in the simulated comparisons.","marker":"[7]"},{"why":"Supplies a generalist policy compared in simulation for both language and goal-image input modalities.","marker":"[43]"},{"why":"Gives the grasp module that computes the initial end-effector pose from the starting keypoint in the real-world setup.","marker":"[18]"},{"why":"Provides the lightweight detector trained as VIEW's keypoint module to extract starting points, waypoints, and endpoints from the sketches.","marker":"[30]"}],"fun_headline_variants":["Draw a sketch, robot executes: 87.5% unseen task success","Sketches beat speech for robot instructions: 87.5% on unseen tasks","Quiet robot commands: hand-drawn sketches hit 87.5% on new tasks","Forget words, doodles direct robots to 87.5% on new tasks","Hand-drawn visual instructions: 87.5% success on unseen robot tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes the vision-language model's Python execution functions stay correct even when the same model's natural-language task and planning outputs are wrong; the paper's own numbers show a 13-billion-parameter model with weak language comprehension achieving high execution success, and that separation is the premise on which the main result rests.","fun_headline_variants_meta":{"raw":{"variants":["Draw a sketch, robot executes: 87.5% unseen task success","Sketches beat speech for robot instructions: 87.5% on unseen tasks","Quiet robot commands: hand-drawn sketches hit 87.5% on new tasks","Forget words, doodles direct robots to 87.5% on new tasks","Hand-drawn visual instructions: 87.5% success on unseen robot tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3861,"prompt_tokens":1041,"completion_tokens":2820,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":2711}},"tokens_in":657,"tokens_out":2820,"duration_ms":17875,"temperature":1.0,"reasoning_tokens":2711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:35:29.032296+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a fixed set of RoVI sketches, then deliberately corrupt the model's natural-language task and planning responses while keeping the generated code and keypoint coordinates unchanged; if execution success drops correspondingly, the claim that execution is unaffected by perception errors is refuted, and if it stays high, keypoints and code are carrying the task.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the base manipulation episodes and images from which the 15K-instance RoVI Book training set is created."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the 13-billion-parameter vision-language model that is fine-tuned on RoVI Book and reports the highest real-world execution success."},{"cited_title":"Octo: An open-source generalist robot policy","cited_arxiv_id":null,"evidence_quote":"Supplies a generalist policy compared in simulation for both language and goal-image input modalities."},{"cited_title":"Ultralytics YOLO, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the lightweight detector trained as VIEW's keypoint module to extract starting points, waypoints, and endpoints from the sketches."}],"review_version":1}