{"id":"f4c15bdc-81d1-44a9-a258-4bd39af1fb1e","arxiv_id":"2505.02166","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CrayonRobo trains a vision-language-action model to read colored 2D prompt overlays (contact point, end-effector axes, movement direction) and output SE(3) contact poses, enabling step-by-step and long-horizon robotic manipulation.","lead":"CrayonRobo lets a robot user draw simple colored marks on a camera image, a dot for where to touch and lines for gripper orientation and movement direction, so a vision-language-action model can execute a manipulation step. The paper shows this approach works across many simulated articulated-object tasks and on several real-world objects, with both hand-drawn and automatically generated prompts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Success is scored by an unstated object-movement threshold that also generates training labels, so demonstrated SE(3) accuracy is not yet separated from the heuristic's ease of producing any motion.","rationale":"The reader identified the same weak spot: heuristic labels plus a binary movement threshold. I agree, and I think this is the most load-bearing concern because it targets the evidence for the central claim itself, not just the comparison set. The prompt design and training losses are coherent, and the real-world transfer attempt is a genuine strength; I am not claiming the method is wrong. But the paper never states the distance threshold, and both training and evaluation use the same 'does the part move' criterion. This creates a plausible alternative explanation for the high success rates: the model may learn to reproduce the heuristic's easy-contact behavior rather than to interpret crayon prompts into precise 3D directions. That is why the verdict should remain CONDITIONAL rather than ACCEPT. The missing RT-Sketch/MOKA/PIVOT comparisons and the self-replicated RT-Trajectory are real weaknesses for the relative claim, but they are secondary; even if those baselines were included, the absolute success-rate claim would still be unverifiable without a quantitative success criterion. Similarly, the 'w/o P' fine-tuning experiment is over-claimed but not central. I would keep the reader's CONDITIONAL verdict; no change is needed beyond emphasizing that the threshold check is the critical revision.","tokens_in":16660,"tokens_out":8299,"duration_ms":110409,"concrete_test":"Re-run the simulation evaluation used for Table 1, first disclosing the exact threshold from Section 4.1, then with stricter task-specific success criteria: (i) the articulated part reaches at least 70% of its full joint range, (ii) predicted contact point and z/y axes are within 2 cm and 15 degrees of the annotated ground truth, and (iii) success requires the correct direction of motion (e.g., drawer slides out, door opens outward). If Ours(s)/Ours(f) advantage over RT-Traj and ManipLLM drops by more than about 0.1 or reverses, the reported success rates overstate the model's pose accuracy. If code or checkpoints are not released, reconstructing the evaluation from the described pipeline with the disclosed threshold is the minimal check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is that the central quantitative evidence for 'accurate SE(3) contact poses and movement directions' depends on a success metric that is both under-specified and shared with the label-generation heuristic. Section 4.1 defines success only as 'thresholding the distance that the object part moves,' without reporting the threshold or a reference distance per task. Section 3.2 records ground-truth poses and directions by 'interacting with objects and recording the success contact pose,' using the same rule-based heuristics. If the threshold is permissive, a model that makes any contact that nudges the part counts as successful even when the predicted pose is off by centimeters or tens of degrees and the post-contact direction is wrong. Because the same weak 'part moved' criterion generates the training labels and the evaluation success, the high numbers in Tables 1 and 3 could reflect the ease of triggering motion rather than prompt-conditioned SE(3) accuracy. The real-world numbers do not fix this: Section 4.4 judges 'If the task has been completed' with no quantitative definition, five trials per task, and author-drawn prompts. The central claim is therefore not yet verifiable from the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CrayonRobo, a vision-language-action (VLA) model that takes RGB images overlaid with color-coded 2D 'crayon' prompts (blue contact point, red z-axis, green y-axis, yellow post-contact moving direction), together with corresponding numerical text prompts, and outputs an SE(3) contact pose and a 3D moving direction. A progressive training scheme supervises the model with text, orthogonality, and projection losses, and long-horizon tasks are decomposed into key-frame prompts executed sequentially. The authors evaluate on SAPIEN/PartNet-Mobility articulated-object tasks with seen/unseen splits against Flowbot3D, ManipLLM, Implicit3D, and a replicated RT-Trajectory baseline, plus real-world Franka experiments with manually drawn and automatically generated prompts, and report favorable success rates.","tokens_in":16857,"tokens_out":9729,"duration_ms":112447,"significance":"If the central claim is supported, CrayonRobo would be a practically useful interface for specifying robotic manipulation goals: it adds explicit directional and contact information to visual prompting, addressing ambiguities of language and over-specification of goal images or videos, and the key-frame decomposition is a sensible way to handle long-horizon tasks. The paper's strengths include the progressive input-pair training strategy, the simple four-color prompt semantics, the automatic (Grounded-DINO plus GPT-4) and manual prompt pipelines, and the breadth of simulator comparisons across many articulated-object categories together with zero-shot real-world transfer. However, the paper does not ship code, and the quantitative claims rest on an underspecified success criterion; until the metric is tightened and direct pose-accuracy numbers are reported, the margin over baselines is difficult to interpret.","major_comments":[{"comment":"The load-bearing evidence for 'accurate SE(3) contact poses and movement directions' is the binary success rate, defined in §4.1 only as 'thresholding the distance that the object part moves.' The threshold value and the reference distance per task are never reported. Since §3.2 generates the training labels with the same rule-based heuristics that define success, a permissive threshold would let any contact that nudges the part count as success even if the predicted pose is off by centimeters or tens of degrees and the post-contact direction is wrong. Please report the threshold in physical units per task or per part, report a success-versus-threshold sweep, and add direct metrics: predicted-versus-GT contact-point position error, orientation error, and moving-direction angular error. These direct metrics are necessary to separate the model's SE(3) accuracy from the heuristic's ease of producing any motion.","section":"§4.1 and §3.2"},{"comment":"The real-world evaluation does not give an independently checkable success definition: §4.4 states only that a trial is successful 'If the task has been completed,' with five trials per task, author-drawn prompts, and no quantitative criterion such as a required joint angle or part displacement. In the 'w/o P.' rows, the model is fine-tuned on the model's own predicted poses from successful trials as 'ground truth,' so these numbers (5/10 and 6/10) measure self-distillation rather than prompt-free generalization with independent labels. Please specify a per-task success criterion, report per-trial outcomes, and use independently measured ground-truth actions (e.g., recorded robot states or human-specified keyframes) for the w/o P. experiment.","section":"§4.4 and Table 3"},{"comment":"There is a contradiction about which point is unprojected to produce the 3D contact location: §3.1 says the model predicts ap'_0 and that this predicted point is mapped to aP0, while §3.4.2 says 'ap0 is projected into 3D space' (the input prompt point). If execution uses the input point, the model's position prediction and the claimed noise-correction behavior in §4.2 are not exercised; if it uses the predicted point, the text is wrong. Clarify this, because it determines whether the model actually outputs an SE(3) contact pose or only a rotation and direction.","section":"§3.4.2 vs §3.1"}],"minor_comments":[{"comment":"The loss weights λ1, λ2, and λ3 are never specified; include their values or a sensitivity study for reproducibility.","section":"§3.3.2"},{"comment":"The discretization description '100 discrete bins [-50,50], with each bin spanning 0.02' is internally inconsistent (a 100-unit range at 0.02 width gives 5,000 bins); clarify the binning of the normalized direction vector.","section":"§3.3.2"},{"comment":"The projection loss compares 2D projections of predicted 3D directions with the input 2D directions, so it enforces 2D–3D projection consistency rather than independently determining the 3D direction; state explicitly that LP is a regularizer and that LT carries the main 3D supervision.","section":"§3.3.2"},{"comment":"The 16 seen-task and 16 unseen-task columns are unlabeled, so it is impossible to see per-task performance; add a task list or column headers.","section":"Table 1"},{"comment":"The statement that adding 'slight noise' to directions perpendicular to the camera improves visibility should be described with a concrete procedure (e.g., noise distribution and magnitude) for reproducibility.","section":"§3.4.1"},{"comment":"The ground-truth text says 'The contact point is at ap0,' but the contact point in ground truth should presumably be a 3D coordinate, not the 2D input prompt; align the notation with §3.1.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main risk is the shared heuristic between label generation and the simulator success metric; if the authors provide absolute thresholds and direct pose-error metrics, the central claim can be made checkable. I do not see circularity in the supervised-imitation setup, and the identified shortcomings are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: CrayonRobo is a real, useful idea wrapped in an under-specified evaluation. The color-coded 2D prompt encoding—blue contact dot, red z-axis, green y-axis, yellow movement direction—goes beyond RT-Sketch and RT-Trajectory by adding orientation and post-contact motion, and the progressive training scheme (text, orthogonality, projection losses) is coherent and well-ablated. The simulator results are strong across many articulated-object tasks, and the real-world zero-shot transfer is a credible proof of concept.\n\nSoft spots, in rough order of importance. First, the success metric in Section 4.1 is \"thresholding the distance that the object part moves\" with no threshold reported and no reference distance per task. Since the same rule-based heuristics generate the training labels in Section 3.2, the reported 0.74/0.72 success rates could partly reflect the ease of nudging a part past a lenient threshold rather than accurate prompt-conditioned SE(3) prediction. This needs the threshold, per-task reference distances, and ideally pose-error metrics (contact point, orientation, direction) so performance is separated from the heuristic's generosity. The real-world results help—tasks like open microwave or heat toaster are naturally categorical—but they are five trials per task with author-drawn prompts.\n\nSecond, the comparison set is thinner than it looks. RT-Sketch, MOKA, and PIVOT are discussed but not evaluated; RT-Trajectory is a self-replication; AVDC is an indirect video-quality check with Gemini. That weakens the claim of advantage over other visual-prompt methods. The self-replication is understandable, but it needs to be flagged as such.\n\nThird, the \"w/o P\" experiment fine-tunes on the model's own successful predictions and then tests without prompts. It is a self-distillation exploration, not a prompt-free ablation, and should be framed that way. Also absent: error bars and trial counts for the simulator numbers.\n\nNone of this sinks the central argument. The paper shows that a VLA can interpret these crayon-style prompts to produce usable SE(3) poses and directions, with qualitative real-world evidence to back it. But the quantitative evidence is not as airtight as the tables suggest. I'd send this to peer review with requests for the missing reporting details and, if practical, at least one comparison against an existing visual-prompt baseline. The idea is novel enough to justify referee time.","headline":"A genuinely new prompt interface for specifying grasp orientation and post-contact motion, but the success metric is under-specified and the closest visual-prompt baselines are missing.","tokens_in":17477,"tokens_out":5420,"would_cite":true,"duration_ms":62440,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Color-coded crayon prompts on an image let a vision-language-action model predict $SE(3)$ contact poses and post-contact motion, chaining key-frames into long-horizon tasks.","keywords":["vision-language-action model","visual prompting","robotic manipulation","SE(3) pose prediction","keyframe planning","articulated objects","sim-to-real transfer","prompt noise tolerance"],"falsifier":"Score the same trained model on held-out objects while measuring object-part motion with an external encoder or motion capture, and require the articulated joint to reach at least half of its full range; if success collapses relative to the paper's binary movement threshold, then the threshold, not the predicted $SE(3)$ poses, is carrying the result. As a second check, compare the predicted contact points and gripper axes against manually annotated ground truth on real objects.","tokens_in":16437,"feed_emoji":"🖍️","tokens_out":8967,"duration_ms":90202,"temperature":0.7,"pith_summary":"Manipulation instructions are usually given as language, goal images, or videos, but language is ambiguous and images or videos carry irrelevant detail. CrayonRobo's proposal is to overlay simple 2D crayon-style marks — a blue dot for the contact point, red and green lines for the gripper's approach axes, and a yellow line for the movement after contact — on an RGB image of the object. A vision-language-action model is trained to read these prompts, together with text giving their numeric 2D coordinates, and to output the corresponding $SE(3)$ contact pose plus an optional 3D moving direction. A sequence of prompted key-frames then specifies the whole plan, so long-horizon tasks are completed step by step. The paper reports higher success than prior prompting, language, and goal-image baselines in simulation, and workable zero-shot performance on five real-world tasks.","feed_headline":"Drawing on a photo tells a robot how to grasp and move","feed_subtitle":"Colored marks specify the contact point, gripper axes, and moving direction, which the model turns into precise 3D actions.","key_machinery":"The core mechanism is the crayon prompt language: four colored geometric marks overlaid on the object image (blue contact dot, red gripper z-axis, green gripper y-axis, yellow post-contact moving direction), each mirrored numerically in the text prompt to remove overlap ambiguity. The training strategy progressively teaches the model to exploit more prompt components, and three losses pin the predictions to geometry: a cross-entropy text-supervision loss over discretized direction bins, a Gram-Schmidt orthogonality loss between the two gripper axes, and a projection loss that unprojects the predicted 3D directions back to 2D and checks them against the input 2D prompts with cosine similarity. At inference the prompts can be drawn by a person or generated automatically by an object detector plus a language model selecting among candidate direction lines.","core_discovery":"The paper claims that a 2D visual-language prompt pair — an image overlaid with colored marks plus a text string containing the same 2D values — is enough for a vision-language-action model to predict an accurate six-degree-of-freedom contact pose and a next-moving direction, and that chaining key-frame prompts completes long-horizon manipulation. The model predicts the contact position in 3D by unprojecting the 2D point with depth and camera parameters, and predicts the gripper's z-axis and y-axis as 3D unit vectors whose orthogonality is enforced, plus an optional post-contact translation direction. The reported experiments show that progressively adding directional prompts raises success rates, that the model tolerates noise in the drawn prompts, and that the same prompted policy transfers to real objects without additional sim-to-real fine-tuning.","pith_inferences":["Beyond the paper, the projection loss couples 2D prompt space to 3D prediction space using only depth and camera intrinsics, so the same consistency check could be applied to other vision-language pose predictors.","Beyond the paper, the prompt language could be extended to rotation-rate or velocity prompts for continuous actions, reusing the two-keyframe trick the paper uses for rotations.","Beyond the paper, the noise tolerance suggests an interactive correction loop: a user sketches a coarse goal, the robot executes, and the user adjusts only the wrong mark, which may be cheaper than redrawing or re-prompting.","Beyond the paper, automatic prompt generation is the noisiest link; feeding the model's own predicted poses back to refine the automatically selected lines could close the loop and reduce the need for manual drawing."],"forward_implications":["A user can specify a manipulation goal by drawing marks on a photo, with no need to write a detailed language instruction or supply a goal image or video.","Because each key-frame is a self-contained prompted sub-goal, long-horizon tasks can be composed from reusable primitives, and per-step success rates compound into overall reliability.","Prompt noise up to 20 percent of the directional values has little effect on simulator success, so non-expert hand-drawn or automatically generated prompts are usable in practice.","The prompted model can be fine-tuned on a specific task and then run without prompts, suggesting a path from prompt-guided training to prompt-free execution.","The same policy works on real objects and multi-step tasks without additional sim-to-real fine-tuning."],"supporting_citations":[{"why":"Supplies the VLA backbone architecture, adapter fine-tuning strategy, and the simulator data-collection style the method follows.","marker":"[32]"},{"why":"Provides the rule-based interaction heuristics used to generate ground-truth contact poses and moving directions in SAPIEN.","marker":"[42]"},{"why":"Supplies the SAPIEN articulated-object simulator and PartNet-Mobility assets used for training and evaluation.","marker":"[55]"},{"why":"The trajectory-sketch prompting baseline the paper replicates and outperforms, motivating directional prompts.","marker":"[20]"},{"why":"The CLIP visual encoder that extracts features from the prompted RGB images.","marker":"[46]"},{"why":"The LLaMA language backbone whose adapters are fine-tuned to output pose and direction text.","marker":"[53]"},{"why":"The LoRA low-rank adaptation method used to keep the language model mostly frozen.","marker":"[21]"},{"why":"Provides the Gram-Schmidt orthogonality loss that keeps predicted gripper axes perpendicular.","marker":"[17]"},{"why":"Detects the object bounding box used to place the contact dot in automatic prompt generation.","marker":"[38]"},{"why":"Selects among candidate direction lines to auto-generate the red, green, and yellow prompts.","marker":"[1]"}],"fun_headline_variants":["Sketch to steer: 2D marks guide robot hands in 3D","Robot reads crayon marks to plan grasps and motions","Draw a dot, robot gets the point: prompt-driven control","Visual doodles become robot commands for manipulation","Crayon-style prompts turn images into robot instructions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the rule-based simulator heuristics produce correct ground-truth contact poses and that moving an object part past the scoring threshold really means the task was accomplished; if either is too permissive, the reported success rates overstate the robot's skill.","fun_headline_variants_meta":{"raw":{"variants":["Sketch to steer: 2D marks guide robot hands in 3D","Robot reads crayon marks to plan grasps and motions","Draw a dot, robot gets the point: prompt-driven control","Visual doodles become robot commands for manipulation","Crayon-style prompts turn images into robot instructions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1212,"prompt_tokens":923,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":206}},"tokens_in":539,"tokens_out":289,"duration_ms":3779,"temperature":1.0,"reasoning_tokens":206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T01:00:22.917070+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the same trained model on held-out objects while measuring object-part motion with an external encoder or motion capture, and require the articulated joint to reach at least half of its full range; if success collapses relative to the paper's binary movement threshold, then the threshold, not the predicted $SE(3)$ poses, is carrying the result. As a second check, compare the predicted contact points and gripper axes against manually annotated ground truth on real objects.","supporting_citations":[{"cited_title":"Manipllm: Embodied multimodal large language model for object-centric robotic manipulation","cited_arxiv_id":null,"evidence_quote":"Supplies the VLA backbone architecture, adapter fine-tuning strategy, and the simulator data-collection style the method follows."},{"cited_title":"Where2act: From pixels to actions for articulated 3d objects","cited_arxiv_id":null,"evidence_quote":"Provides the rule-based interaction heuristics used to generate ground-truth contact poses and moving directions in SAPIEN."},{"cited_title":"Chang, Leonidas J","cited_arxiv_id":null,"evidence_quote":"Supplies the SAPIEN articulated-object simulator and PartNet-Mobility assets used for training and evaluation."},{"cited_title":"The loss of orthogonality in the gram-schmidt orthogonalization process","cited_arxiv_id":null,"evidence_quote":"Provides the Gram-Schmidt orthogonality loss that keeps predicted gripper axes perpendicular."}],"review_version":1}