{"id":"80cab641-fd53-4cd4-8751-381bee95bdc4","arxiv_id":"2412.09286","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A language-to-video-to-action pipeline lets a robot learn novel manipulation skills from generated demonstration videos, roughly tripling zero-shot task success in MetaWorld.","lead":"DemoGen is a robot-learning pipeline that turns a plain-language instruction into training videos, then extracts actions from those videos to teach an imitation policy. In simulated MetaWorld tasks, policies trained on these generated demonstrations succeeded on about three times as many novel tasks as the same policies trained on existing demonstrations from other tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim hinges on IDM transfer to distorted generated videos, but the paper never measures IDM accuracy on those videos, while its own numbers suggest expected accuracy near chance.","rationale":"The reader's weakest_assumption is exactly the IDM transfer from expert videos to generated videos, and my independent check of the numbers strengthens it. Table 3 reports zero-shot task accomplishment of only 63.2% for generated videos; Appendix A.2 admits roughly 40% erroneous generations for novel tasks. If the downstream policy is trained on all generated demonstrations without filtering, a large fraction of the data is non-expert. The paper does not describe any quality filter before IDM extraction (Algorithm 2 has none), and it does not evaluate IDM accuracy on generated videos. Given IDM zero-shot accuracy of 76.0-81.6% on clean expert videos, the pipeline's success depends on two unmeasured links. This is not an internal inconsistency; the reported numbers are compatible with a model that filters implicitly and a 'three times' result that survives only on tasks where the IDM happens to transfer well. But the argument as written does not establish which. I therefore keep the reader's CONDITIONAL verdict: the central claim is plausible but not yet supported without the missing measurement. I would require the IDM-on-generated-videos accuracy plus per-task seed-level results with a video filter; a single table change could either settle or refute the concern. If the authors provided the missing IDM evaluation and it came back comparable to Table 5 numbers, I would accept the central claim as supported.","tokens_in":9823,"tokens_out":1873,"duration_ms":18112,"concrete_test":"Recover the exact videos from the zero-shot fold used to produce Table 4, run the trained IDM on them, and compare its predicted actions against the expert actions from the corresponding MetaWorld task. Report per-task IDM accuracy on generated videos alongside the per-task accomplishment rate of the same videos (human-rated or task-success-rated). If IDM accuracy on generated videos is below roughly 76% or the task-success-filtered accuracy is not above expert-video IDM accuracy, the zero-shot policy gain in Table 4 cannot be attributed to correct action extraction, and the central claim requires a filtering or relabeling step that is not present.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim is that language-generated videos can replace expert demonstrations for zero-shot skill learning. The load-bearing link is the IDM (Section 2.1.3, Eq. 5): it must convert DVG-generated video frames into correct actions. The paper reports IDM zero-shot accuracy only on expert-collected videos (Appendix Table 5: 76.0-81.6%); it never reports IDM accuracy on the videos actually used by the ILM. Under the paper's own metrics, DVG generations for novel tasks fail to accomplish the task in 36.8% of cases (Table 3: 63.2% accomplishment) and are erroneous in roughly 40% of cases (Appendix A.2). If a generated video does not accomplish the task, the IDM's correct action labels from the video are by definition not expert actions for that task. Two consequences follow. First, at least ~37% of generated demonstrations encode non-expert behavior; injecting these into the ILM should degrade, not triple, performance unless a filtering mechanism exists, and none is described (Appendix A.2 only suggests a 'designed validation module' for future work). Second, even on 'accomplishing' generated videos, the IDM is trained on 20 trajectories per few-shot task with distractors like varying textures and lighting in the generated videos (preprocessing noise, edge extraction); with reported zero-shot IDM accuracy as low as 76% on clean expert videos, action-label noise on out-of-distribution generated frames could push accuracy toward chance for multi-action tasks. The paper's zero-shot result depends on the unverified conjunction that (i) enough generated videos are task-correct and (ii) the IDM labels them accurately. Since neither quantity is measured on the operational distribution, the 'nearly three times' claim is not secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DemoGen is a four-module pipeline for learning robot manipulation skills from natural language instructions. A vision-language model expands task descriptions, a fine-tuned text-to-video diffusion model (DVG) generates demonstration videos, an inverse dynamics model (IDM) converts video frames into state-action pairs, and an imitation learning model (ILM, LCBC or RT-1) learns a policy by behavioral cloning. The paper evaluates the pipeline on MetaWorld tasks under a two-fold split, comparing policies trained on expert demonstrations with policies trained on generated demonstrations. The headline claim is that in the zero-shot setting, generated demonstrations yield 'nearly three times the task accomplishment rates' of expert-data baselines (6/8 and 5/8 versus 2/8 in Table 4). The paper also reports human evaluation of generated video quality and an appendix with failure cases and an IDM generalization study.","tokens_in":10051,"tokens_out":8841,"duration_ms":80301,"significance":"If the central claim were fully supported, DemoGen would be a meaningful step toward replacing expert demonstrations and environment interaction with text-only inputs for novel manipulation skills. The modular design is sensible, the use of existing components (GLM-4, Tune-A-Video, IDM, LCBC, RT-1) makes the approach accessible, and the inclusion of a two-fold split, human evaluation, and explicit failure cases is a strength. However, as it stands the evidence is not sufficient: the IDM's ability to label the generated videos, which is the key transfer point, is never measured, and the headline comparison is based on binary task-level checkmarks without statistical basis. These are correctable with additional experiments and reporting, rather than fundamental flaws in the pipeline concept.","major_comments":[{"comment":"The headline claim of 'nearly three times the task accomplishment rates' rests on Table 4, which reports only binary checkmarks over eight task categories with no number of evaluation episodes, seeds, variance, confidence intervals, or statistical test. A change from 2/8 to 6/8 for LCBC and 2/8 to 5/8 for RT-1 is not sufficient to establish a threefold improvement. The authors should report per-task success rates with multiple seeds and mean/std, clarify what a checkmark denotes (for example, any successful episode versus a success-rate threshold), and explain how the eight columns map to the 22 tasks described in §3.1.","section":"§3.4.2, Table 4"},{"comment":"The load-bearing assumption is that the IDM trained on expert demonstrations from few-shot tasks can label actions in DVG-generated videos for zero-shot tasks. Appendix Table 5 reports IDM accuracy only on expert-collected videos and under training configurations (5/45 tasks, 50/100 trajectories) that do not match the main setting (20 trajectories per few-shot task). Meanwhile Appendix A.2 states that roughly 40% of novel-task generations are erroneous, Table 3 gives a 63.2% human-rated accomplishment rate for zero-shot generated videos, and no validation/filtering module is implemented ('a designed validation module' is future work). Without measuring IDM action-label accuracy on the exact generated videos used in Table 4, the downstream policy results cannot be attributed to correct state-action extraction. The authors should add this measurement or implement filtering and report results on the filtered data.","section":"§2.1.3, Eq. (5), Algorithm 2, Appendix A.2"},{"comment":"The text states that in the few-shot setting the generated demonstrations achieved performance comparable to expert data, but Table 4 shows RT-1 dropping from 8/8 with expert data to 4/8 with generated data, while LCBC changes from 5/8 to 6/8. This is a substantial degradation for one of the two ILMs and contradicts the claim that 'the quality of the generated demonstrations is similar to the collected manipulations.' The authors should explain this discrepancy or soften the few-shot comparability claim.","section":"§3.4.2, Table 4"}],"minor_comments":[{"comment":"The text says performance 'slightly decreased under the few-shot learning setting,' but Table 3 shows few-shot learning with higher scores on all three criteria (88.3/92.5/96.1) than zero-shot learning (57.9/63.2/71.6); the sentence should refer to the zero-shot setting.","section":"§3.4.1"},{"comment":"The phrase 'Visualization results in Table 3' is incorrect because Table 3 is the human-evaluation table; the reference should likely be to Fig. 3 or Fig. 5.","section":"§3.4.1"},{"comment":"Algorithm 2, titled 'Novel skill scheme,' ends after DVG video generation and omits the IDM action-extraction and ILM training steps described in §2.2 and §2.3; these steps should be included for the algorithm to match the pipeline description.","section":"Algorithm 2"},{"comment":"The human evaluation relies on four evaluators; reporting inter-annotator agreement or individual evaluator scores would strengthen the reliability of the demonstration-quality metrics.","section":"§3.3.1, Table 3"},{"comment":"The VLM is the closed-source GLM-4-0520; for reproducibility the authors should specify the exact API version, prompt templates, and sampling parameters, since small prompt differences can change the generated demonstrations.","section":"§3.2.3"},{"comment":"There are minor typographical issues in the introduction, including 'enbles' and 'assiting'; these should be corrected in a revised manuscript.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an ICLR 2025 camera-ready; for a journal-style review the appropriate recommendation is major_revision. The main concerns are empirical completeness (IDM accuracy on generated videos, seed-level statistics) rather than any flaw in the derivation. The authors should also correct the internal contradictions between the text and Tables 3 and 4 before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The specific combination here — VLM prompt expansion, a Tune-A-Video-based generator, an inverse dynamics model, and imitation learning — is new, and the few-shot results show that generated data can sometimes stand in for expert demos. The appendix is also honest about a roughly 40% generation failure rate on novel tasks, which is more than most papers admit. That said, the central claim does not hold up under inspection.\n\nThe biggest problem is the zero-shot comparison. Baselines are trained only on few-shot expert demonstrations and then evaluated on zero-shot tasks, so they have no training data for those tasks at all. DemoGen, on the other hand, gets generated demonstrations for the zero-shot tasks. Of course it does better. The 'nearly three times' result is really 'some data beats no data,' not 'generated data beats expert data.' The few-shot setting is the fair comparison, and there the picture is mixed: LCBC does fine, but RT-1 drops from 8/8 to 4/8.\n\nThe stress-test point is also valid. The IDM's accuracy is reported only on expert-collected videos (76–81.6% zero-shot). The actual pipeline feeds it generated videos, which on novel tasks fail 36.8% of the time. The paper never measures IDM accuracy on those generated videos, so the link between generation quality and policy learning is simply unverified. The appendix even concedes that a filtering module would be needed, but none is used.\n\nThere are smaller issues: Table 4 is binary checkmarks with no seeds or variance, and Section 3.4.1 says performance 'slightly decreased' under few-shot learning while Table 3 shows few-shot scores are much higher. That's a clear internal contradiction. No code or data is released.\n\nWho is this for? People working on imitation learning from synthetic data, or anyone who wants a concrete example of the full pipeline in MetaWorld. It deserves a serious referee, but the zero-shot experiment needs redesign and the IDM transfer needs measurement. If you cite it, cite it as a pipeline proposal, not as evidence that language-generated demos work. If you ever review a revision, ask for per-seed results, IDM accuracy on generated videos, and a zero-shot comparison that includes expert demos for the same tasks.","headline":"DemoGen is a sensible pipeline and worth knowing about, but the headline zero-shot result is not secured by the experiments as written.","tokens_in":742,"tokens_out":1000,"would_cite":false,"duration_ms":36463,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a natural-language instruction alone can supply the training data for a novel robot manipulation skill: generated demonstration videos, labeled by an inverse dynamics model, yield zero-shot task accomplishment rates…","keywords":["robot skill learning","language-conditioned imitation","demonstration generation","video diffusion model","inverse dynamics model","zero-shot task acquisition","behavioral cloning","natural language instructions"],"falsifier":"Measure the IDM's action-prediction accuracy directly on DVG-generated videos of zero-shot tasks, comparing predicted actions against ground-truth actions recorded by the simulator. If that accuracy is near chance or markedly below the 76.0-81.6% reported for expert-collected videos, the pipeline's action labels are unreliable and the downstream policy gain cannot be attributed to the generated demonstrations.","tokens_in":9567,"feed_emoji":"🤖","tokens_out":8014,"duration_ms":69081,"temperature":0.7,"pith_summary":"DemoGen is a four-stage pipeline that tries to let a robot learn a brand-new manipulation skill from nothing but a natural-language instruction. A vision-language model expands the instruction into a detailed prompt, a text-to-video diffusion model renders demonstration videos of the task, an inverse dynamics model (IDM) converts those frames into state-action pairs, and an imitation learning policy maps states to actions. On a standard multi-task manipulation benchmark, policies trained on these generated demonstrations achieve roughly three times the task accomplishment rate of policies trained on expert-collected demonstrations when tested on tasks never seen during training (the zero-shot setting). If this holds, language alone could become a usable source of robot training data, removing the need for human teleoperation or environment exploration per new task.","feed_headline":"Generated demo videos nearly triple zero-shot robot task success","feed_subtitle":"DemoGen turns language instructions into demonstration videos and labeled actions, letting robots learn novel tasks without expert demos.","key_machinery":"The load-bearing object is the inverse dynamics model (IDM), a transformer-encoder-plus-vision-transformer network that predicts the action at time $t$ from a sliding window of $n+1$ observed states; it is what turns generated pixels into a policy-training dataset. Around it sit the video diffusion generator, a latent text-to-video model fine-tuned with pose conditioning through an adapter, and a behavioral cloning objective that minimizes the squared error between predicted actions and the IDM-extracted actions.","core_discovery":"The central claim is that high-fidelity, task-consistent demonstration videos generated by a fine-tuned video diffusion model can serve as a substitute for human-collected expert demonstrations when learning novel skills. Given only a short task description, the pipeline expands it with a vision-language model, generates videos conditioned on text and rendered pose trajectories, and applies an IDM trained on a small number of expert episodes to label each frame sequence with actions. The resulting state-action dataset trains language-conditioned behavioral cloning policies; on unseen tasks from the benchmark, the generated data gives policies around three times the accomplishment rate of the same policies trained on expert data, with one of the evaluated policies succeeding on six of eight zero-shot tasks from generated data versus two of eight from expert data.","pith_inferences":["A reader may infer that the IDM is the bottleneck: if it labels imperfect generated videos poorly, the threefold zero-shot gain would shrink, and a validation filter that discards the roughly 40% erroneous generations would likely widen the gap.","The reported trend that more training tasks help IDM generalization more than more trajectories per task suggests the pipeline would improve faster by broadening IDM training diversity than by collecting more demonstrations per task.","Because human raters judged only 63.2% of zero-shot generated videos as accomplishing the task, testing policies trained only on human-approved videos versus all generated videos would isolate how much of the gain comes from video fidelity versus IDM robustness."],"forward_implications":["A robot can be given a novel task and receive a training signal without any expert demonstration of that task, reducing data-collection labor and teleoperation cost.","Zero-shot skill acquisition becomes a matter of prompt quality: richer language expansions and better video generations should raise downstream policy success without additional robot interaction.","The IDM's ability to label generated videos controls the ceiling of the pipeline, so improvements in IDM generalization should transfer directly into higher accomplishment rates.","Any task expressible as a natural-language description becomes a candidate for automatic demonstration generation, so task coverage can grow by adding prompts rather than by collecting episodes."],"supporting_citations":[{"why":"Supplies the multi-task manipulation benchmark, the 22 tasks, and the ground-truth states and actions used for collecting expert episodes and evaluating task accomplishment.","marker":"(Yu et al., 2020)"},{"why":"Provides the improved denoising diffusion process (IDDPM) that underlies the forward and reverse passes of the video generator.","marker":"(Nichol & Dhariwal, 2021)"},{"why":"Supplies the Tune-A-Video text-to-video architecture that the demonstration video generator fine-tunes with spatio-temporal attention.","marker":"(Wu et al., 2023)"},{"why":"Provides the T2I-Adapter used for pose-conditioned video generation, keeping the robot arm's motion consistent with the intended trajectory.","marker":"(Mou et al., 2023)"},{"why":"The vision-language model used for prompt expansion, turning short task descriptions into detailed prompts that guide video generation.","marker":"(GLM et al., 2024)"},{"why":"Provides one of the two imitation learning policies compared on expert versus generated demonstrations.","marker":"(Stepputtis et al., 2020)"},{"why":"Provides the transformer-decoder-based imitation policy that serves as the other baseline for skill learning.","marker":"(Brohan et al., 2022)"}],"fun_headline_variants":["Language-generated demos triple robot success","From text to training: demos triple novel task rates","Video diffusion demos boost robot learning 3x","Robot skills from language: demo videos triple success","No expert needed: language demos triple task success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the inverse dynamics model, trained on only 20 expert demonstrations per few-shot task, assigns correct action labels to video frames of zero-shot tasks it has never seen, even though roughly 40% of those generated videos are acknowledged to be erroneous and human raters judged only 63.2% of zero-shot videos as accomplishing the task.","fun_headline_variants_meta":{"raw":{"variants":["Language-generated demos triple robot success","From text to training: demos triple novel task rates","Video diffusion demos boost robot learning 3x","Robot skills from language: demo videos triple success","No expert needed: language demos triple task success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1255,"prompt_tokens":841,"completion_tokens":414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":341}},"tokens_in":457,"tokens_out":414,"duration_ms":3854,"temperature":1.0,"reasoning_tokens":341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:26.182944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the IDM's action-prediction accuracy directly on DVG-generated videos of zero-shot tasks, comparing predicted actions against ground-truth actions recorded by the simulator. If that accuracy is near chance or markedly below the 76.0-81.6% reported for expert-collected videos, the pipeline's action labels are unreliable and the downstream policy gain cannot be attributed to the generated demonstrations.","supporting_citations":[{"cited_title":"Improved denoising diffusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Provides the improved denoising diffusion process (IDDPM) that underlies the forward and reverse passes of the video generator."},{"cited_title":"Language-conditioned imitation learning for robot manipulation tasks","cited_arxiv_id":null,"evidence_quote":"Provides one of the two imitation learning policies compared on expert versus generated demonstrations."}],"review_version":1}