{"id":"4b416b1c-1f9c-4d76-93d4-bab7c616bcfc","arxiv_id":"2411.09052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ClevrSkills provides a 33-task, 330k-trajectory benchmark showing that vision-language robot policies struggle to compose base manipulation skills into novel long-horizon tasks.","lead":"ClevrSkills is a new benchmark for compositional reasoning in robot manipulation, with 33 tasks and 330k demonstrations generated in the ManiSkill2 simulator. The paper finds that several open-source vision-language models fail to compose learned base skills into more complex tasks, highlighting a gap in current robotics policies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot L1/L2 failures may reflect unreliable L0 skills rather than compositional failure; conditional skill-success check needed.","rationale":"The reader's conditional verdict is appropriate. The benchmark artifact is substantial and independently useful: 33 tasks, oracle policies, 330k trajectories, and rich annotations. The core negative claim, however, depends on isolating compositional failure from low-level execution failure. The reported L0 results do not establish that base skills are reliable enough to support that isolation, and the arithmetic of chaining imperfect primitives already predicts very low L1/L2 completion rates. The proposed conditional-skill-success or oracle-substitution test would directly settle whether the near-zero zero-shot results are due to composition or execution. The factual inconsistency in §5.3 versus Table 3 is secondary but should be fixed. Since the main concern is the same as the reader's weakest assumption and the conditional verdict already accommodates it, no verdict change is needed.","tokens_in":22830,"tokens_out":5674,"duration_ms":62688,"concrete_test":"Use the existing predicate/key-step logging to compute per-sub-goal success during zero-shot L1/L2 rollouts. Restrict the success metric to episodes in which all prerequisite L0 sub-goals (e.g., pick the specified object, place it on the target) were completed at the moment attempted. If conditional success remains near zero, the compositional claim is supported; if it rises substantially, the reported failure is an artifact of unreliable base skills. A stronger variant: replay each such rollout while substituting oracle solver actions for the low-level skill segments, leaving the model's high-level selection fixed, and re-score the task.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract and §7) is that current VLMs fail at compositional generalization. The evidence is the near-zero zero-shot success on L1/L2 tasks (Table 3). This inference is load-bearing on the premise that L0 base skills are reliable enough that task failure can be attributed to composition. §5.3 and Table 2 show this premise is not met: the best L0 success is 62.91% on seen attributes and 55.41% on unseen attributes, with particularly weak per-task results on pick, place, push, and rotate (Figure 5). Since §5.3 states that L1 and L2 tasks require roughly 9 and 11 successful L0-skill executions respectively, even a flawless compositional planner would complete only a small fraction of long episodes. For illustration, if each primitive succeeds 63% of the time, nine sequential primitives succeed with probability approximately 1.6%, which is in the same regime as the observed near-zero L1 success. The reported failures are therefore quantitatively consistent with low-level skill unreliability alone, independent of any compositional deficit. The paper's phrase 'reasonable performance on the L0 base skills' (§5.3) is not supported by the numbers. A secondary factual issue: the §5.3 sentence 'only Octo achieves non-zero success rate on any of the tasks' contradicts Table 3, where JAT shows 0.83% L2 zero-shot success; this should be corrected.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ClevrSkills, a benchmark built on ManiSkill2 with 33 manipulation tasks organized into three levels of compositional difficulty (L0 base skills, L1 intermediate compositions, L2 complex compositions), together with a dataset of roughly 330k oracle-generated trajectories with multi-modal prompts, language annotations, bounding boxes, key steps, and dense rewards. The authors benchmark several open-source vision-language policies (JAT, Octo, RoboFlamingo, and their own StreamRoboLM) under zero-shot and fine-tuned conditions. The central claim is that current VLMs, even after pre-training on large amounts of data, fail at compositional generalization in robotics: they can partially acquire L0 skills but cannot compose them into novel L1/L2 tasks.","tokens_in":23070,"tokens_out":3544,"duration_ms":36625,"significance":"The benchmark infrastructure is a genuine contribution: a three-level curriculum explicitly designed around skill composition, a large dataset with rich annotations, oracle solvers with motion planning, and a public release commitment. The evaluation is clean in the sense that the oracle policies are scripted and the models are measured empirically, so there is no circular derivation or fitted-parameter concern. The per-task results and the comparison across four baselines are useful. However, the central negative claim—that observed L1/L2 failures demonstrate a compositional generalization deficit—is currently confounded by incomplete mastery of L0 skills, as detailed in the major comments. If that confound is addressed with a conditional analysis, the benchmark would be a valuable resource for the robotics and VLM communities.","major_comments":[{"comment":"The central negative claim—that failures at L1/L2 reflect compositional generalization failure—is not supported by the current evidence because the prerequisite L0 skills are not reliably mastered. Table 2 shows that the best L0 success is 62.91% (StreamRoboLM-OPT, seen attributes) and 55.41% (StreamRoboLM-Llama3, unseen attributes), and Figure 5 shows particularly low per-task success on pick, place, push, rotate, and touch variants. Since §5.3 states that L1 and L2 tasks require roughly 9 and 11 successful L0-skill executions respectively, even a flawless compositional planner with per-primitive success probability p ≈ 0.63 would complete a 9-primitive L1 episode with probability approximately 0.63^9 ≈ 1.6%, which is the same regime as the reported near-zero L1 success. The observed L1/L2 failures are therefore quantitatively consistent with low-level skill unreliability alone, independent of any compositional deficit. The phrase 'reasonable performance on the L0 base skills' in §5.3 is not supported by the reported numbers. Please add a conditional analysis that reports L1/L2 success only on episodes in which the requisite L0 sub-skills were individually executed successfully, or otherwise control for L0 failure modes before attributing failure to composition.","section":"§5.3, Table 2, Figure 5"},{"comment":"The sentence 'only Octo [43] achieves non-zero success rate on any of the tasks' is contradicted by Table 3: in the L2 zero-shot columns, JAT reports 0.83% success and Octo reports 0.83%; in the L1 zero-shot columns, Octo reports 0.33%. Please correct the text to agree with the table, e.g., by stating that almost all zero-shot L1/L2 success rates are at or near zero, with Octo and JAT occasionally reporting small non-zero values.","section":"§5.3, Table 3"}],"minor_comments":[{"comment":"The file name 'succes.npy' appears to be a typo; it should be 'success.npy'.","section":"Appendix B.1"},{"comment":"The text refers to 'RTT Connect', which should be 'RRT-Connect' (the rapidly-exploring random tree connect variant used by OMPL).","section":"Appendix G"},{"comment":"Prompt (b) for the Place task reads 'Put object with tex:object 1 texture on object with tex:object2 texture' and appears to have missing braces around the placeholders; it should likely be 'Put object with {tex:object}1 texture on object with {tex:object}2 texture'.","section":"Appendix A.1, Task 4"},{"comment":"The description of Stack topple contains the typo 'compostion' instead of 'composition'.","section":"Appendix A.3, Task 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is well matched to a datasets-and-benchmarks venue and the infrastructure appears original. The main issue is interpretational rather than infrastructural: the zero-shot L1/L2 results need a conditional or per-primitive analysis before the compositional-failure conclusion can be drawn. I would encourage the authors to add that analysis, correct the internal inconsistency in §5.3, and clarify how the benchmark can separate 'skill not acquired' from 'skills not composed'."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the ClevrSkills artifact is real and worth having, but the paper's main claim—that current VLMs fail compositional reasoning in robotics—is not supported by the experiments as run, because base skills are too unreliable to attribute failures to composition. There is also a factual slip in §5.3.\n\nWhat's new: this is the first manipulation benchmark I know of that combines VIMA-style multi-modal prompts with true low-level end-effector control, a three-level task curriculum, and 330k oracle trajectories with rich annotations. The environment extends ManiSkill2, so it is usable. The task design (L0 skills, L1 compositions, L2 long-range compositions) is thoughtful, and the oracle solvers are scripted and well documented. For the community, this is a useful resource for measuring closed-loop compositional generalization in robot policies.\n\nThe soft spots are real. The abstract and conclusion claim that models 'fail on compositional reasoning,' but the evidence is the near-zero zero-shot success on L1/L2. That inference depends on the L0 skills being reliable enough. The best L0 success is 62.91%; some skills (pick, place, push, rotate) are much worse per Figure 5. With roughly 9-11 sequential L0 executions per L1/L2 task, a model with 63% per-skill success would complete an episode with probability around 1.6%, even with a perfect planner. So the observed near-zero results are quantitatively consistent with weak low-level skills alone. The paper's phrase 'reasonable performance on the L0 base skills' (§5.3) does not hold up. A conditional success analysis—e.g., evaluate L1 success only on episodes where the model successfully executed the required L0 sub-skills—would disentangle this. That is the main fix I would ask for.\n\nAlso, §5.3 states 'only Octo achieves non-zero success rate on any of the tasks' in the zero-shot setting, but Table 3 reports JAT at 0.83% on L2. That is an internal contradiction and should be corrected. Minor: results are reported without error bars; with 20 seeds, a standard deviation would help.\n\nWho should care: anyone evaluating robot foundation models or designing compositional benchmarks. The dataset and environment are the contribution; the experimental interpretation is currently too strong.\n\nMy recommendation: send to peer review. The benchmark is useful and the negative result is interesting, but the paper should be revised to add the conditional analysis (or explicitly acknowledge the confound) and fix the factual error. I would be happy to see it accepted after that.","headline":"Useful benchmark artifact, but the central claim about compositional failure is undercut by unreliable L0 base skills; needs a conditional-success analysis.","tokens_in":23618,"tokens_out":4065,"would_cite":true,"duration_ms":36045,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Even after pretraining, vision-language models fail to compose robot skills into new tasks.","keywords":["compositional generalization","vision-language models","robot manipulation","benchmark suite","imitation learning","multi-modal prompts","ManiSkill2","pick-and-place"],"falsifier":"Train a policy to near-perfect success (for example, above 95%) on every L0 skill separately, then evaluate it zero-shot on L1 and L2 tasks. If such a policy solves the composed tasks at high rates, the paper's conclusion that current models fail at compositional generalization would be overturned; if it still fails, the composition bottleneck would be confirmed. A simpler check would be to rerun the paper's evaluation while conditioning only on episodes where the model completed all L0 subskills, and see whether L1/L2 success rises.","tokens_in":22630,"feed_emoji":"🤖","tokens_out":5259,"duration_ms":51981,"temperature":0.7,"pith_summary":"This paper asks whether vision-language models that have learned basic robot manipulation skills can recombine those skills to solve new, higher-level tasks without being trained on them. To answer it, the authors build ClevrSkills, a simulation benchmark with 33 tasks arranged in three levels: simple motor skills (L0), intermediate compositions such as sorting and stacking (L1), and complex compositions such as balancing a scale or sorting by throwing (L2). They generate 330,000 oracle trajectories with language, visual, and multi-modal annotations, and they benchmark several open vision-language policies. The central finding is that even after pre-training on large task collections and after fine-tuning on the benchmark's own data, all tested models fail on the composed L1 and L2 tasks, with near-zero zero-shot success and low fine-tuned success. If correct, the result indicates that current vision-language-action models do not exhibit compositional generalization in embodied manipulation, and that progress will require more than scaling data or fine-tuning on the component skills.","feed_headline":"Vision-language models fail to compose robot skills","feed_subtitle":"New ClevrSkills benchmark: zero-shot success on composed L1/L2 tasks is near zero even after fine-tuning.","key_machinery":"The load-bearing mechanism is the three-level task curriculum built on predicate-based task definitions. Physical predicates (such as EEAtPos, OnTop, Inside, Touch) specify target states, and logical predicates (Set, Sequence, Once) combine them into tasks, so every L1 and L2 task is explicitly a composition of L0-level skills and can be solved by scripted oracle 'solver' policies that recursively invoke lower-level solvers. This design makes it possible to train on L0, then test zero-shot on L1 and L2, and to know exactly which skill compositions are required. The accompanying dataset provides 330k trajectories, hierarchical language annotations, bounding boxes, key-step frames, and multi-modal prompts. The oracle trajectories both define the demonstration data and act as a ceiling for evaluating the learned policies.","core_discovery":"The paper's central claim is that state-of-the-art vision-language models, despite broad pre-training and additional fine-tuning, cannot compose previously learned low-level manipulation skills to perform novel high-level tasks in a closed-loop robotics setting. The evidence is the benchmark's L0->L1->L2 curriculum: models trained on L0 reach at best roughly 63% success on those simple skills, yet their zero-shot success on L1 and L2 tasks is essentially zero, and fine-tuning on L1 and L2 leaves most success rates in single digits. The paper also shows that multi-modal (text-plus-image) prompts are harder for several baselines than text-only prompts, suggesting part of the difficulty lies in grounding visual task specifications. The authors conclude that compositional generalization in robotics remains unsolved by current VLM-based policies.","pith_inferences":["Because the best model only masters about 63% of L0 skills, part of the L1/L2 failure could be downstream of unreliable base skills rather than compositional reasoning; a policy with near-perfect L0 performance would be needed to isolate the composition deficit.","If the bottleneck is compositional, then collecting more demonstration data for component skills alone will not suffice; methods that explicitly train on predicate-level composition or symbolic planning with learned skill modules would be a natural testable next step.","The same benchmark design could be extended to test whether hierarchical policies that separately learn skills and a high-level selector outperform end-to-end video-conditioned policies.","A real-robot counterpart would be needed to see whether the failure persists outside simulation, since sim-to-real transfer may add new confounds."],"forward_implications":["If the finding holds, evaluating compositional generalization in robotics requires controlling for base-skill mastery; raw task success on composed tasks cannot separate planning failures from execution failures.","Current VLM-based policies trained on large internet and robotics datasets will not generalize zero-shot to novel task specifications that recombine known skills.","Fine-tuning on the composed tasks themselves does not close the gap, so the failure is not merely a matter of seeing the task distribution; the models lack a general composition mechanism.","The benchmark's predicate structure lets future work measure progress per skill and per composition, rather than with a single end-to-end metric.","Multi-modal prompts add difficulty beyond text-only instructions, so visual grounding of task specifications is itself a bottleneck."],"supporting_citations":[{"why":"Provides the ManiSkill2 simulator and robot/controller definitions that ClevrSkills extends.","marker":"[16]"},{"why":"Prior VIMA benchmark that introduced multi-modal prompts and compositional probing; ClevrSkills extends it to low-level end-effector control.","marker":"[20]"},{"why":"CLEVR, the inspiration for controlled compositional visual reasoning, adapted here to embodied tasks.","marker":"[21]"},{"why":"RoboFlamingo, a VLM-based imitation baseline evaluated on ClevrSkills.","marker":"[28]"},{"why":"Octo, a generalist policy baseline evaluated on ClevrSkills.","marker":"[43]"},{"why":"JAT, a generalist transformer agent baseline evaluated on ClevrSkills.","marker":"[12]"},{"why":"LRR model, the video-reasoning architecture underlying StreamRoboLM.","marker":"[4]"},{"why":"Flamingo, whose cross-attention design is used in StreamRoboLM.","marker":"[1]"}],"fun_headline_variants":["VLMs can't compose robot skills, even after fine-tuning","ClevrSkills: zero-shot composition on L1/L2 is near zero","Fine-tuned VLMs still fail at skill composition","Compositional generalization in robotics: VLMs fail","Robot skill composition remains unsolved for VLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion assumes that near-zero success on L1 and L2 tasks reflects an inability to compose skills, not the models' incomplete mastery of the L0 skills themselves; since the best model reaches only about 63% success on L0, a composed task can fail simply because a base skill misfires.","fun_headline_variants_meta":{"raw":{"variants":["VLMs can't compose robot skills, even after fine-tuning","ClevrSkills: zero-shot composition on L1/L2 is near zero","Fine-tuned VLMs still fail at skill composition","Compositional generalization in robotics: VLMs fail","Robot skill composition remains unsolved for VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001231,"raw_usage":{"total_tokens":5054,"prompt_tokens":939,"completion_tokens":4115,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":4032}},"tokens_in":555,"tokens_out":4115,"duration_ms":29404,"temperature":1.0,"reasoning_tokens":4032,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:06:48.157437+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a policy to near-perfect success (for example, above 95%) on every L0 skill separately, then evaluate it zero-shot on L1 and L2 tasks. If such a policy solves the composed tasks at high rates, the paper's conclusion that current models fail at compositional generalization would be overturned; if it still fails, the composition bottleneck would be confirmed. A simpler check would be to rerun the paper's evaluation while conditioning only on episodes where the model completed all L0 subskills, and see whether L1/L2 success rises.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ManiSkill2 simulator and robot/controller definitions that ClevrSkills extends."},{"cited_title":"Jiang, A","cited_arxiv_id":null,"evidence_quote":"Prior VIMA benchmark that introduced multi-modal prompts and compositional probing; ClevrSkills extends it to low-level end-effector control."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RoboFlamingo, a VLM-based imitation baseline evaluated on ClevrSkills."}],"review_version":1}