Pith. sign in

REVIEW 4 major objections 6 minor 59 references

Towards a Generalizable Bimanual Foundation Policy via Flow-based Video Prediction

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CogRobot claims that a pretrained text-to-video model, fine-tuned first to predict optical flow and then to predict video from that flow, can act as the planning layer for a dual-arm robot, with a goal-conditioned diffusion policy…

desk verdict A genuinely novel two-stage flow idea that the experiments don't yet validate: no closed-loop flow ablation, no data-efficiency measurement. read the letter →

arxiv 2505.24156 v1 pith:2WA744AR submitted 2025-05-30 cs.CV cs.RO

Chenyou Fan , Fangzheng Yan , Chenjia Bai , Jiepeng Wang , Chi Zhang , Zhen Wang , Xuelong Li This is my paper
classification cs.CVcs.RO
keywords bimanualmanipulationtext-to-videogenerationopticalflowdiffusionpolicyvideopredictionvision-language-actionmodelsrobotfoundationmodelimitationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a generalizable bimanual robot policy can be built without training a vision-language-action model from scratch or inventing a unified action space. It fine-tunes a pretrained text-to-video diffusion model on a limited amount of dual-arm data, with optical flow as an intermediate representation: a text-to-flow module predicts future motion fields from the initial image and a language instruction, and a flow-to-video module turns those fields into future frames. A lightweight goal-conditioned diffusion policy then treats each predicted frame as a target observation and outputs joint actions for the two arms. The authors report that this flow-guided two-stage pipeline improves generated-video quality over direct fine-tuning of the same base model, and that it outperforms a single-view diffusion policy on real dual-arm tasks. The payoff, if the claim holds, is that language-conditioned video prediction grounded in optical flow can supply the planning layer for bimanual control without large action-labeled datasets.

What carries the argument

The central object is optical flow used as an intermediate variable between language and video. For a frame pair, flow is the per-pixel displacement field mapping each pixel in the initial observation to its location in a future observation; the paper converts each flow field into an RGB flow video via magnitude-and-angle color coding, so the pretrained video VAE can encode it without training a new flow autoencoder. The text-to-flow model learns the distribution over flow videos conditioned on the initial image and instruction, and the flow-to-video model learns the distribution over future RGB videos conditioned on the initial image, instruction, and flow video, by concatenating flow latents with noisy video latents. This two-stage decomposition is what carries the argument: flow supplies explicit motion cues that text alone cannot.

What would settle it

On a fixed bimanual task with a trained CogRobot policy, condition the goal-reaching diffusion policy on ground-truth future frames from recorded demonstrations for some rollouts and on frames produced by the text-to-flow and flow-to-video cascade for others, keeping the start states identical; if success is comparable, the predicted frames are faithfully carrying the plan, whereas a large drop would show the cascade breaks the link between planning and executable actions. A cheaper check is to measure the average endpoint error of the predicted optical flow and the fraction of predicted goal frames whose implied joint configurations are outside the robot's reachable workspace.

Watch

Extended reading notes

Core claim

The paper's central claim is CogRobot: a bimanual foundation policy obtained by fine-tuning the pretrained text-to-video model CogVideoX-2B in two stages. The first stage, text-to-flow, predicts a sequence of optical-flow maps between the initial observation and future frames, which concretizes the motion implied by the language instruction; the second stage, flow-to-video, synthesizes the future RGB frames conditioned on those flow maps. A separate goal-conditioned Diffusion Policy, trained on real observation pairs, converts the predicted frames into executable dual-arm joint positions. The key claim is that optical flow works as an intermediate physical representation that lets the video model capture fine-grained arm and object motion with limited bimanual data, reducing language ambiguity and yielding trajectories that can actually be executed. Evidence includes improved PSNR, SSIM, LPIPS, and FVD over directly fine-tuned video models and higher real-world success rates than a single-view diffusion policy on lift-bag and pull-box tasks.

Load-bearing premise

The method depends on generated video frames being close enough to physically reachable states that the goal-conditioned diffusion policy, trained only on real observation pairs, can act on them successfully; the paper does not measure flow prediction error, end-to-end video quality, or the distribution shift between predicted and real goal frames.

Editorial extensions

If this is right

  • Fine-tuning a leading text-to-video model on bimanual data, with flow as an intermediate, can produce a dual-arm planner without training a VLA from scratch or defining a unified action space.
  • Predicting flow before pixels improves generated trajectory-video quality on the RDT and RoboMIND validation sets compared with directly fine-tuning the same base model.
  • The full CogRobot pipeline outperforms a single-view Diffusion Policy on two real dual-arm tasks, and the improvement is larger on the multi-stage pull-box task than on the simpler lift-bag task.
  • The conclusion states a limitation: each downstream task still needs its own separate goal-conditioned diffusion policy to turn predicted videos into actions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the video planning layers are agnostic to the robot's action space: because actions never enter the text-to-flow or flow-to-video models, the same trained video predictor could in principle serve different embodiments or control interfaces, with only the low-level goal-conditioned policy retrained.
  • A direct testable extension is to ablate the flow intermediate at three levels — no flow, ground-truth flow, and predicted flow — and compare downstream task success; this would separate the value of flow as a representation from the cost of flow prediction errors.
  • The claimed data-efficiency benefit could be quantified by training on progressively smaller subsets of the bimanual demonstrations and measuring both video quality and task success, rather than the fixed 100-demonstration setup reported.
  • Because the real-world evaluation covers two tasks and one camera-based baseline, the generalizability claim would be tested more sharply by adding unseen object arrangements, viewpoint changes, and a point-cloud or multi-view baseline operating on the same single camera input.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes CogRobot, a bimanual manipulation policy built by fine-tuning the text-to-video model CogVideoX-2B in two stages: a text-to-flow stage that predicts visual optical-flow videos from the current RGB frame and language instruction, and a flow-to-video stage that synthesizes future RGB frames conditioned on the predicted flow. A lightweight goal-conditioned Diffusion Policy is then trained to map the initial observation and a predicted future frame to low-level joint actions for the two arms. The method is evaluated in the RoboTwin simulation benchmark on four tasks and on two real-world bimanual tasks with a custom Vision Pro teleoperation system, comparing against Diffusion Policy, DP3, and RDT. The paper also reports offline video-generation quality metrics (PSNR, SSIM, LPIPS, FVD) on RDT and RoboMIND validation sets. The central claims are that optical-flow guidance reduces the data requirement for fine-tuning T2V models and that the resulting video prediction improves downstream task success relative to action-only policies.

Significance. If the central claims hold, the paper would make a useful contribution to robot learning by showing that a strong pre-trained text-to-video model can be adapted to bimanual manipulation through a physically meaningful intermediate representation (optical flow), without training a VLA model from scratch or requiring a unified action space. The idea of decomposing trajectory prediction into text-to-flow and flow-to-video is sensible and plausibly reduces the ambiguity of language-to-video mapping. The paper also contributes a real dual-arm teleoperation setup and a small real-world dataset. However, the evidence provided is insufficient to establish the two load-bearing claims: (i) that the two-stage flow mechanism is responsible for the reported downstream gains, and (ii) that the method significantly reduces robot-data requirements. The task-success experiments do not include a closed-loop baseline using the directly fine-tuned text-to-video model with the same diffusion policy, and no data-efficiency experiment is reported. The real-world results are based on 20 trials per task without confidence intervals.

major comments (4)
  1. [§5.4, Table 3] The video-quality ablation is not sufficient to validate the two-stage flow mechanism. Table 3 compares CogVideoX-2B-Flow against SFT variants on PSNR/SSIM/LPIPS/FVD, but the paper does not state whether the flow input to the flow-to-video model is ground-truth flow from FlowFormer++ or flow predicted by the text-to-flow model. In deployment (§5.3) the flow-to-video model is conditioned on predicted flow, so the reported improvements may not reflect the actual cascade error from Stage 1. The paper should report end-to-end video quality with predicted flow, and ideally also report the flow-prediction error itself.
  2. [§5.2, §5.3, §5.4] The central claim that flow-guided video prediction improves downstream task success is not supported by a closed-loop ablation. All task-success experiments in Tables 1 and 2 compare the full CogRobot pipeline against action-only baselines (DP, DP3, RDT); there is no comparison of CogVideoX-2B-SFT (direct text-to-video fine-tuning) paired with the same goal-conditioned diffusion policy. Without this control, the improvements over DP could come from the video-prediction component in general rather than from the optical-flow intermediate. Please add a closed-loop baseline with direct SFT video prediction and the same goal-conditioned policy.
  3. [Abstract, §1, §5] The claim that the method "significantly reduces the robot-data requirement" is asserted but never directly measured. No experiment varies the amount of training demonstrations for the flow-based model versus the direct SFT baseline, and no data-efficiency curve is provided. Given that the method trains two video models plus a diffusion policy, this claim needs quantitative support. Please include a data-scaling study on at least one simulation task, comparing CogRobot against CogVideoX-2B-SFT with the same downstream policy.
  4. [Table 2, §5.3] The real-world results are reported as average success rates over 20 trials without confidence intervals or statistical significance tests. With 20 trials, the difference between 0.70 and 0.50 (Lift Bag) is within sampling variability, and the large gap on Pull Box (0.75 vs 0.05) is encouraging but should be accompanied by uncertainty quantification. Furthermore, the deployment loop feeds predicted video frames as goals to a policy trained on real observation pairs; the paper does not analyze the distribution shift between real and generated goal frames. Please report confidence intervals and, if possible, a measure of goal-frame distribution shift or a closed-loop comparison using ground-truth goal frames.
minor comments (6)
  1. [§1] The sentence "Bimanual manipulation represents a pivotal for embodied agents" is missing a noun; it should be "a pivotal challenge" or similar.
  2. [Table 1] The table layout is confusing because the column headers "Put Apple Cabinet Block Handover" and "Pick Apple Messy Container Place" span two columns each; please reformat so each task has its own column with a clear header. The very large standard deviation reported for DP3 (Point Cloud) on Put Apple (74.7±42.2) also warrants a note or a different aggregation.
  3. [Table 3] The video-quality metrics are reported without error bars or multiple-seed variation. Since the RDT and RoboMIND validation sets are large, please report standard deviations or at least state whether the differences are significant.
  4. [Fig. 2, §5.4] The ablation label "CogVideoX-2B-Flow (ours)" in Table 3 and Fig. 2 should be made consistent with the main method name "CogRobot" or explicitly defined to avoid confusion about whether the full system or just the video predictor is being evaluated.
  5. [§2 Preliminaries] In the POMDP definition, the reward function R: S x A -> {0,1} is described as measuring whether the language goal l is achieved, but l is not an argument of R in the notation; please clarify the dependency.
  6. [References] In §2 Related Work, the text refers to "Sora [16]" but reference [16] is the Open-Sora paper; please correct the citation to the actual Sora reference or adjust the text to match the cited work.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the two-stage flow-video predictor and the goal-conditioned diffusion policy are trained on external data with no fitted quantity renamed as a prediction.

full rationale

CogRobot's claimed derivation chain is empirical rather than algebraic: a text-to-flow model learns a distribution over optical-flow videos, a flow-to-video model learns a distribution over robot videos conditioned on flow, and a goal-conditioned diffusion policy is trained on real observation-action pairs. None of these components is defined in terms of another component's output in a way that would make a reported result true by construction. Optical-flow supervision is obtained from an external estimator (FlowFormer++) applied to ground-truth videos, and the downstream policy is trained on real goal observations, not on the model's own predicted frames. The video-quality comparison in Table 3 is a standard held-out evaluation against SFT baselines, and the task-success comparisons in Tables 1 and 2 are against external baselines on external benchmarks. The only self-citation in the paper, reference [39], appears in a related-work enumeration of diffusion-based action prediction methods and is not load-bearing for any design choice or experimental claim. Concerns about whether Table 3 uses oracle flow rather than predicted flow, and about the missing closed-loop ablation of the flow intermediate, are legitimate evaluation gaps but do not constitute circularity: no prediction reduces to a fitted parameter or to an input by definition.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several unvalidated domain assumptions (single-view sufficiency, flow estimator reliability, generative model transfer, and generated frames as goals) and on a set of hand-picked hyperparameters; no new physical entities are introduced.

free parameters (5)
  • video length (17 frames) = 17
    Fixed video length for both flow and video prediction; chosen by hand, not justified.
  • resolution = 256x256
    All videos resized to 256x256; chosen for computational convenience.
  • low optical flow magnitude filtering threshold = not specified
    C.2 states frames with low optical flow magnitude are filtered out, but the threshold is never given, making the training data composition ambiguous.
  • goal step range nmax for diffusion policy = not specified
    Eq. (4) samples n from U(1, nmax), but nmax is never defined.
  • fine-tuning step counts = 20k/30k/15k/5k
    Training durations for the two stages across datasets were chosen empirically and not ablated.
assumptions (4)
  • domain assumption Single external RGB camera view provides sufficient state information to plan and execute bimanual manipulation
    Section 2 formulates the task as a POMDP with observation ot as RGB image from an external camera; the whole pipeline conditions on this single view.
  • domain assumption Optical flow estimated by FlowFormer++ is an adequate ground-truth motion target for fine-tuning
    Section 5.1 and C.2 use FlowFormer++ to generate ground-truth optical flow; errors in flow estimation are treated as ground truth.
  • domain assumption The pre-trained text-to-video model CogVideoX retains sufficient world knowledge after domain fine-tuning on limited bimanual data
    Section 3.1 assumes the generative prior transfers to robotic video; failure cases in Fig. 2 show this transfer is imperfect.
  • domain assumption A generated video frame can be treated as a reachable goal observation for the low-level policy
    Section 3.3 explicitly states each frame in the predicted video can be treated as a target observation; this is the key assumption behind the control loop.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards a Generalizable Bimanual Foundation Policy via Flow-based Video Prediction." pith.science (2026). https://pith.science/paper/2WA744AR

@misc{pith2026250524156,
  author       = {Pith},
  title        = {Pith review of: Towards a Generalizable Bimanual Foundation Policy via Flow-based Video Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WA744AR}},
  note         = {Machine review of arXiv:2505.24156}
}
read the original abstract

Learning a generalizable bimanual manipulation policy is extremely challenging for embodied agents due to the large action space and the need for coordinated arm movements. Existing approaches rely on Vision-Language-Action (VLA) models to acquire bimanual policies. However, transferring knowledge from single-arm datasets or pre-trained VLA models often fails to generalize effectively, primarily due to the scarcity of bimanual data and the fundamental differences between single-arm and bimanual manipulation. In this paper, we propose a novel bimanual foundation policy by fine-tuning the leading text-to-video models to predict robot trajectories and training a lightweight diffusion policy for action generation. Given the lack of embodied knowledge in text-to-video models, we introduce a two-stage paradigm that fine-tunes independent text-to-flow and flow-to-video models derived from a pre-trained text-to-video model. Specifically, optical flow serves as an intermediate variable, providing a concise representation of subtle movements between images. The text-to-flow model predicts optical flow to concretize the intent of language instructions, and the flow-to-video model leverages this flow for fine-grained video prediction. Our method mitigates the ambiguity of language in single-stage text-to-video prediction and significantly reduces the robot-data requirement by avoiding direct use of low-level actions. In experiments, we collect high-quality manipulation data for real dual-arm robot, and the results of simulation and real-world experiments demonstrate the effectiveness of our method.

Figures

Figures reproduced from arXiv: 2505.24156 by the authors.

Figure 1
Figure 1. Overview of CogRobot. (a) We build a dual-arm robotic system based on Realman arms [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Visualization of different models. (a) Videos are generated directly using the CogVideoX [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Attention maps across different models. The results are obtained using same initial [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visualization of the predicted video (blue dashed box) and the actual execution trajectory [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of the generated flow and predicted video in the RoboTwin simulation [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the real-world experimental setup and tasks. (a) We develop a VR-based [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Architecture of the flow-to-video model. The bimanual video and the flow video are [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: More visualization of the generated flow and predicted video in the RoboTwin simulation [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: More visualization of the generated flow and predicted video for the real-world Realman [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 25 canonical work pages

  1. [1]

    Peract2: Benchmarking and learning for robotic bimanual manipulation tasks

    Markus Grotz, Mohit Shridhar, Yu-Wei Chao, Tamim Asfour, and Dieter Fox. Peract2: Benchmarking and learning for robotic bimanual manipulation tasks. InCoRL 2024 Workshop on Whole-body Control and Bimanual Manipulation: Applications in Humanoids and Beyond, 2024

    2024

  2. [2]

    Robotwin: Dual-arm robot benchmark with generative digital twins (early version).arXiv preprint arXiv:2409.02920, 2024

    Yao Mu, Tianxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo. Robotwin: Dual-arm robot benchmark with generative digital twins (early version).arXiv preprint arXiv:2409.02920, 2024

    arXiv 2024

  3. [3]

    Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn

    Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. InProceedings of Robotics: Science and Systems, 2023

    work page 2023

  4. [4]

    Sukhatme

    I-Chun Arthur Liu, Sicheng He, Daniel Seita, and Gaurav S. Sukhatme. V oxact-b: V oxel-based acting and stabilizing policy for bimanual manipulation. In8th Annual Conference on Robot Learning, 2024

    work page 2024

  5. [5]

    Stabilize to act: Learning to coordinate for bimanual manipulation

    Jennifer Grannen, Yilin Wu, Brandon Vu, and Dorsa Sadigh. Stabilize to act: Learning to coordinate for bimanual manipulation. InConference on Robot Learning, pages 563–576. PMLR, 2023

    work page 2023

  6. [8]

    Taco: Benchmarking generalizable bimanual tool-action-object understanding

    Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21740–21751, 2024

    2024

  7. [9]

    Towards human-level bimanual dexterous manipulation with reinforcement learning.Advances in Neural Information Processing Systems, 35:5150–5163, 2022

    Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning.Advances in Neural Information Processing Systems, 35:5150–5163, 2022

    work page 2022

  8. [10]

    Bi-touch: Bimanual tactile manipulation with sim-to-real deep reinforcement learning.IEEE Robotics and Automation Letters, 8(9):5472–5479, 2023

    Yijiong Lin, Alex Church, Max Yang, Haoran Li, John Lloyd, Dandan Zhang, and Nathan F Lepora. Bi-touch: Bimanual tactile manipulation with sim-to-real deep reinforcement learning.IEEE Robotics and Automation Letters, 8(9):5472–5479, 2023

    work page 2023

Show all 59 references
  1. [11]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  2. [12]

    π0: A vision-language-action flow model for general robot control, 2024.URL https://arxiv

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control, 2024.URL https://arxiv. org/abs/2410.24164, 2024

    2024 arXiv

  3. [13]

    Rdt-1b: a diffusion foundation model for bimanual manipulation.arXiv preprint arXiv:2410.07864, 2024

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation.arXiv preprint arXiv:2410.07864, 2024

    2024 arXiv

  4. [14]

    Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    2025 arXiv

  5. [15]

    Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025

    Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.arXiv preprint arXiv:2503.06669, 2025

    2025 arXiv

  6. [17]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan.Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an ex...

    2025

  7. [18]

    Open-television: Teleoperation with immersive active visual feedback

    Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, and Xiaolong Wang. Open-television: Teleoperation with immersive active visual feedback. In8th Annual Conference on Robot Learning, 2024

    2024

  8. [19]

    Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation.arXiv preprint arXiv:2412.13877, 2024

    Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation.arXiv preprint arXiv:2412.13877, 2024

    2024 arXiv

  9. [20]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. InACM SIGGRAPH 2024 Conference Papers, pag...

    2024

  10. [21]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020

    2020

  11. [22]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, page 02783649241273668, 2023

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  12. [23]

    Learning manipulation by sequencing motor primitives with a two-armed robot

    Rudolf Lioutikov, Oliver Kroemer, Guilherme Maeda, and Jan Peters. Learning manipulation by sequencing motor primitives with a two-armed robot. InIntelligent Autonomous Systems 13: Proceedings of the 13th International Conference IAS-13, pages 1601–1611. Springer, 2016

    2016

  13. [24]

    A system for imitation learning of contact-rich bimanual manipulation policies

    Simon Stepputtis, Maryam Bandari, Stefan Schaal, and Heni Ben Amor. A system for imitation learning of contact-rich bimanual manipulation policies. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11810–11817. IEEE, 2022

    2022

  14. [25]

    Deep imitation learning for bimanual robotic manipulation.Advances in neural information processing systems, 33:2327–2337, 2020

    Fan Xie, Alexander Chowdhury, M De Paolis Kaluza, Linfeng Zhao, Lawson Wong, and Rose Yu. Deep imitation learning for bimanual robotic manipulation.Advances in neural information processing systems, 33:2327–2337, 2020

    2020

  15. [26]

    Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation.arXiv preprint arXiv:2403.10506, 2024

    Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation.arXiv preprint arXiv:2403.10506, 2024

    2024 arXiv

  16. [27]

    Bigym: A demo-driven mobile bi-manual manipulation benchmark

    Nikita Chernyadev, Nicholas Backshall, Xiao Ma, Yunfan Lu, Younggyo Seo, and Stephen James. Bigym: A demo-driven mobile bi-manual manipulation benchmark. In8th Annual Conference on Robot Learning,

  17. [28]

    Zhao, and Chelsea Finn

    Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile ALOHA: Learning bimanual mobile manipulation using low-cost whole-body teleoperation. In8th Annual Conference on Robot Learning, 2024

    2024

  18. [29]

    Mimicgen: A data generation system for scalable robot learning using human demonstrations

    Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. InConference on Robot Learning, pages 1820–1864. PMLR, 2023

    2023

  19. [30]

    Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning

    Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Fan, and Yuke Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025

    2025

  20. [31]

    Bi-kvil: Keypoints-based visual imitation learning of bimanual manipulation tasks

    Jianfeng Gao, Xiaoshu Jin, Franziska Krebs, Noémie Jaquier, and Tamim Asfour. Bi-kvil: Keypoints-based visual imitation learning of bimanual manipulation tasks. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16850–16857. IEEE, 2024

    2024

  21. [32]

    Interactive imitation learning of bimanual movement primitives.IEEE/ASME Transactions on Mechatronics, 2023

    Giovanni Franzese, Leandro de Souza Rosa, Tim Verburg, Luka Peternel, and Jens Kober. Interactive imitation learning of bimanual movement primitives.IEEE/ASME Transactions on Mechatronics, 2023

    2023

  22. [33]

    InterACT: Inter-dependency aware action chunking with hierarchical attention transformers for bimanual manipulation

    Andrew Choong-Won Lee, Ian Chuang, Ling-Yuan Chen, and Iman Soltani. InterACT: Inter-dependency aware action chunking with hierarchical attention transformers for bimanual manipulation. In8th Annual Conference on Robot Learning, 2024. URLhttps://openreview.net/forum?id=lKGRPJFPCM. 11

    2024

  23. [34]

    Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    2024 arXiv

  24. [35]

    Spatialvla: Exploring spatial representations for visual-language-action model

    Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830, 2025

    2025 arXiv

  25. [36]

    Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024

    2024 arXiv

  26. [37]

    Dexgraspvla: A vision-language-action framework towards general dexterous grasping.arXiv preprint arXiv:2502.20900, 2025

    Yifan Zhong, Xuchuan Huang, Ruochong Li, Ceyao Zhang, Yitao Liang, Yaodong Yang, and Yuanpei Chen. Dexgraspvla: A vision-language-action framework towards general dexterous grasping.arXiv preprint arXiv:2502.20900, 2025

    2025

  27. [38]

    Mole-vla: Dynamic layer-skipping vision language action model via mixture-of-layers for efficient robot manipulation.arXiv preprint arXiv:2503.20384, 2025

    Rongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Dan Wang, Yuan Du, and Shanghang Zhang. Mole-vla: Dynamic layer-skipping vision language action model via mixture-of-layers for efficient robot manipulation.arXiv preprint arXiv:2503.20384, 2025

    2025 arXiv

  28. [39]

    Learning an actionable dis- crete diffusion policy via large-scale actionless video pre-training

    Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. Learning an actionable dis- crete diffusion policy via large-scale actionless video pre-training. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  29. [40]

    Video diffusion models.Advances in Neural Information Processing Systems, 35:8633–8646, 2022

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models.Advances in Neural Information Processing Systems, 35:8633–8646, 2022

    2022

  30. [41]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    2022 arXiv

  31. [42]

    Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023

    2023

  32. [43]

    Tenenbaum

    Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=Mhb5fpA1T0

    2024

  33. [44]

    Grounding video models to actions through goal conditioned exploration

    Yunhao Luo and Yilun Du. Grounding video models to actions through goal conditioned exploration. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview. net/forum?id=G6dMvRuhFr

    2025

  34. [45]

    Robodreamer: Learning compositional world models for robot imagination

    Siyuan Zhou, Yilun Du, Jiaben Chen, YANDONG LI, Dit-Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. InInternational Conference on Machine Learning, 2024. URLhttps://openreview.net/forum?id=kHjOmAUfVe

    2024

  35. [46]

    Vidman: Exploiting implicit dynamics from video diffusion model for effective robot manipulation.Advances in Neural Information Processing Systems, 37:41051–41075, 2024

    Youpeng Wen, Junfan Lin, Yi Zhu, Jianhua Han, Hang Xu, Shen Zhao, and Xiaodan Liang. Vidman: Exploiting implicit dynamics from video diffusion model for effective robot manipulation.Advances in Neural Information Processing Systems, 37:41051–41075, 2024

    2024

  36. [47]

    Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024

    2024 arXiv

  37. [48]

    Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0

    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In2024 IEEE Internati...

    2024

  38. [49]

    Irasim: Learning interactive real-robot action simulators.arXiv preprint arXiv:2406.14540, 2024

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real-robot action simulators.arXiv preprint arXiv:2406.14540, 2024

    2024 arXiv

  39. [50]

    Avid: Adapting video diffusion models to world models.arXiv preprint arXiv:2410.12822, 2024

    Marc Rigter, Tarun Gupta, Agrin Hilmkil, and Chao Ma. Avid: Adapting video diffusion models to world models.arXiv preprint arXiv:2410.12822, 2024

    2024 arXiv

  40. [51]

    AdaWM: Adaptive world model based planning for autonomous driving

    Hang Wang, Xin Ye, Feng Tao, Chenbin Pan, Abhirup Mallik, Burhaneddin Yaman, Liu Ren, and Junshan Zhang. AdaWM: Adaptive world model based planning for autonomous driving. InInternational Conference on Learning Representations, 2025. 12

    2025

  41. [52]

    Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation

    Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation. InProceedings of the IEEE/CVF conference on computer vision and pa...

    2023

  42. [53]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.arXiv preprint arXiv:2403.03954, 2024

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.arXiv preprint arXiv:2403.03954, 2024

    2024 arXiv

  43. [54]

    Image quality metrics: Psnr vs

    Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010

    2010

  44. [55]

    Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    2004

  45. [56]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018

    2018

  46. [57]

    Towards accurate generative models of video: A new metric & challenges.arXiv preprint arXiv:1812.01717, 2018

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges.arXiv preprint arXiv:1812.01717, 2018

    2018 arXiv

  47. [58]

    Bunny-visionpro: Real-time bimanual dexterous teleoperation for imitation learning.arXiv preprint arXiv:2407.03162, 2024

    Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, Shiqi Yang, Ruihan Yang, Xiaojuan Qi, and Xiaolong Wang. Bunny-visionpro: Real-time bimanual dexterous teleoperation for imitation learning.arXiv preprint arXiv:2407.03162, 2024

    2024 arXiv

  48. [59]

    Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning.arXiv preprint arXiv:2406.08858, 2024

    Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning.arXiv preprint arXiv:2406.08858, 2024

    2024 arXiv

  49. [60]

    Using apple vision pro to train and control robots, 2024

    Younghyo Park and Pulkit Agrawal. Using apple vision pro to train and control robots, 2024

    2024

  50. [61]

    Grasp the rope on the box and pull together to bring the box closer

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019. 13 A Details of Data Collection Platform SettingsTo address embodiment gaps encountered when training with Internet bimanual data and to bette...

    2019

  51. [2024]

    URLhttps://openreview.net/forum?id=EM0wndCeoD

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.