Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7B vision-language model trained on code synthesized from in-the-wild videos generates executable policy code for robotic manipulation in a zero-shot manner, outperforming GPT-4o on RLBench, LIBERO, and real-world tasks.

desk verdict Video2Code is a genuinely useful data-curation idea, but the SOTA claim needs confidence intervals and a RoboCodeX baseline before I'd bet on it. read the letter →

arxiv 2501.04268 v1 pith:IWEJWISG submitted 2025-01-08 cs.RO cs.CV

Senwei Xie , Hongyu Wang , Zhanqi Xiao , Ruiping Wang , Xilin Chen This is my paper · ORCID
classification cs.ROcs.CV
keywords roboticmanipulationzero-shotgeneralizationpolicycodegenerationvision-languagemodelvideo-to-codesynthesisasrobotfoundation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RoboPro aims to show that a single open 7B vision-language model can generate executable policy code for robotic manipulation in a zero-shot manner, with no fine-tuning on the target robot or environment. The paper's central claim is that training on 115k code samples synthesized from in-the-wild robot videos lets RoboPro outperform the much larger GPT-4o by 11.6% on RLBench and 17.4% on LIBERO, and reach 72.5% average success on eight real-world tasks. If true, this matters because it replaces expensive runtime code collection and manual simulation annotations with an automatic video-to-code pipeline, potentially scaling robot foundation models from video data alone. The paper also claims the learned skill knowledge transfers to renamed or refactored API formats and to entirely unseen skill sets.

What carries the argument

The load-bearing mechanism is Video2Code, an automatic two-stage data curation pipeline. Stage one uses Gemini-1.5-Flash on 16 key frames per video to extract a concise robot-centric natural-language plan; stage two feeds that plan, the original instruction, and API definitions to DeepSeek-Coder-V2 to generate executable policy code. The API library itself is the second piece: it splits into perception modules, which ground objects and physical properties via GroundingDINO and AnyGrasp, and control modules, which provide heuristic paths for sliding, sweeping, pouring, wiping, and joint rotation. Because RoboPro is trained to call these APIs through code, the authors argue the model internalizes atomic skills independently of their names or signatures. The model is a SigLIP-L vision encoder connected through a two-layer MLP to CodeQwen-1.5-7B, trained in three stages, the last being supervised fine-tuning on the 115k Video2Code samples mixed with general visual instruction data.

What would settle it

Inspect the synthesized training corpus: execute or statically analyze the 115k Video2Code programs against the API library in a simulator and count how many run without error and implement the stated instruction. A finding that the large majority are non-executable or semantically wrong, while RoboPro still scores well downstream, would undercut the claim that it learned procedural knowledge from video-derived code; a clearer version is to train RoboPro on the same data with API-call order scrambled and check whether the RLBench success rate collapses to the no-Video2Code baseline.

Watch

Extended reading notes

Core claim

RoboPro is a vision-language model that takes a wrist-camera RGB-D image, a free-form instruction, and an API library definition, and outputs Python code that calls atomic perception and control functions to produce trajectories. The authors' discovery is that operational videos, not robot demonstrations annotated with code, can serve as the source of training data: Video2Code uses a draft VLM to turn 50k DROID videos into step-by-step natural-language plans, then a code-specialized LLM translates the plans into executable policy code against the API library, yielding 115k aligned image-instruction-code triples. Fine-tuned on this data, RoboPro generates code whose logical ordering of API calls is substantially more accurate than that of GPT-4o, which the authors identify as the main source of its success on RLBench and LIBERO. The authors frame this as evidence that procedural knowledge implicit in videos is transferable to novel tasks, environments, and skill definitions.

Load-bearing premise

The central assumption is that the 115k programs synthesized by Gemini and DeepSeek-Coder are executable and logically correct enough to teach RoboPro real procedural skill; the paper does not verify or filter these programs before training, so if most are flawed the reported gains could come from API and prompt design rather than from video-derived knowledge.

Editorial extensions

If this is right

  • Zero-shot policy code generation from a 7B open model can exceed the manipulation success of a frontier proprietary VLM on both RLBench and LIBERO, without any task-specific training.
  • The same API library and code-format prompts work across two simulators and a real Franka arm, so code generation transfers across embodiments and environments.
  • RoboPro's success on RLBench rises as the fraction of Video2Code data grows from 25% to 100%, indicating that scaling automatically synthesized code data is a viable route to improve such models.
  • Because performance on unseen task-specific APIs (Water Plants, Hit Ball, Scoop Cube) stays above GPT-4o, the procedural knowledge learned from videos transfers to skill sets never seen in training.
  • Removing Video2Code data collapses RLBench success from 42.7% to 0.4%, showing the video-derived code data is the main driver of the gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if Video2Code scales as claimed, any large video corpus containing manipulation, including human-hand or egocentric video, could in principle be distilled into policy-code training data, potentially removing the robot-specific data bottleneck for embodied foundation models.
  • Beyond the paper, because the paper does not verify or filter the synthesized code before training, a direct test of the mechanism is whether RoboPro trained on API-call orders that have been deliberately scrambled loses its advantage; if it does not, the gains may reflect API-format familiarity rather than learned procedural sequencing.
  • Beyond the paper, the method depends on a hand-designed API library, so each new embodiment or sensor suite likely needs a new library; an implicit next step is to learn or adapt the API definitions themselves from data.
  • Beyond the paper, the authors point toward navigation as a next domain, and the same recipe of video-to-plan-to-code could be tested there with minimal changes to the pipeline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes RoboPro, a vision-language model that observes wrist-camera RGB-D images and a free-form instruction, and outputs Python policy code to be executed through a predefined API library of perception and control modules. To obtain training data, the authors introduce Video2Code, a pipeline that uses Gemini-1.5-Flash to convert 50,000 DROID videos into natural-language plans and DeepSeek-Coder-V2 to translate those plans into 115,000 code snippets; these are combined with general visual-instruction data for supervised fine-tuning of CodeQwen-1.5-7B with a SigLIP vision encoder. The paper evaluates RoboPro on nine RLBench tasks, eight LIBERO tasks, and eight real-world tasks, reporting that it outperforms GPT-4o by 11.6% on RLBench and 17.4% on LIBERO, and that it is robust to API renaming, API refactoring, and unseen task-specific skills.

Significance. If the reported results hold, the contribution is significant: Video2Code offers a scalable way to convert in-the-wild videos into policy-code training data, and RoboPro would demonstrate that a 7B open vision-language model can generate executable policy code that generalizes zero-shot across simulators and a real robot, outperforming a proprietary generalist VLM. The paper also provides an early study of robustness to API format changes and unseen skill sets. However, the current evidence is not yet sufficient to support the state-of-the-art claim: the reported margins are within statistical noise for the sample sizes, the closest prior method (RoboCodeX) is not benchmarked, and the synthetic training code is not validated for executability or correctness.

major comments (4)
  1. [Section IV-A.1 and Tables I, II, IV] All results are point estimates from 25 (RLBench), 30 (LIBERO), or 10 (real-world) episodes, with no confidence intervals, standard errors, or multiple seeds. On RLBench, a single episode changes a task average by 4 percentage points, so the headline gap of 11.6% (42.7 vs 31.1) is within roughly two standard errors of the difference, and several per-task gaps are 4-8 points. The statements that RoboPro "significantly outperforms" GPT-4o (Section IV-A.2 and the Abstract) are therefore not supported by the reported statistics. Please provide Wilson intervals or other confidence intervals, run multiple seeds, or perform paired episode-level comparisons when the same episodes are used across methods.
  2. [Section IV-A and Section II-B] The paper identifies RoboCodeX as the closest prior multimodal policy-code generation method, and says the primary comparison group is code-generation methods, yet RoboCodeX is not evaluated in any of Tables I-IV. Without a same-protocol comparison under the same API library, the claims of state-of-the-art zero-shot performance (Abstract and Section IV-A.1) are not established. The authors should add a RoboCodeX comparison, or explicitly justify why it cannot be run under the same evaluation protocol and soften the state-of-the-art claim accordingly.
  3. [Section III-B and Table V] Section III-B reports collecting 115k runtime code data via Gemini-1.5-Flash and DeepSeek-Coder-V2, but no executability check, syntactic validation, or task-success verification is reported for these programs. The ablation in Table V (42.7 vs 0.4 on RLBench when Video2Code is removed) is consistent with the model learning API syntax and naming, since the no-Video2Code model never saw the API library during fine-tuning; it does not establish that the synthetic programs encode correct procedural knowledge. Please provide quantitative quality statistics for the 115k programs (e.g., syntax validity, API-call coverage, human or automated correctness judgments) and include a control trained on non-executable or API-skeleton-only code to separate API familiarity from procedural learning.
  4. [Section IV-B.1 and Section IV-C] The real-world evaluation uses 10 tests per task, so per-task differences such as 80 vs 70 or 60 vs 60 are not meaningful without error bars or a significance test. Likewise, the API-renaming and API-refactoring results in Table I are single-run RLBench averages of 42.7 and 40.4, a 2.3-point gap that is within noise for 25 episodes per task. These robustness claims should either be presented with uncertainty quantification or be described as preliminary observations rather than definitive evidence of robustness.
minor comments (6)
  1. [Section IV-A] The phrase "Our primary comparison tagets" contains a typo and should read "Our primary comparison targets."
  2. [Tables I and II] The table captions contain the garbled phrase "GREYED ON NEED SUPERVISED TRAINING" and "GREYED ON"; these should read "GREYED OUT: REQUIRES SUPERVISED TRAINING" or similar.
  3. [Appendix V-D] The sentence "the results on tow simulation platforms" should read "the results on two simulation platforms."
  4. [Listings 1-3] The prompt text contains the typo "excecute" (also as "Begin to excecute the task"); it should be "execute."
  5. [Section IV-B.1] The phrase "changes in API formation" should be "changes in API format."
  6. [Figure 4] The error breakdown lacks definitions, counts, and an annotation procedure, which makes it difficult to verify the claim that RoboPro reduces logical errors relative to GPT-4o.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claim is an empirical zero-shot evaluation against external benchmarks, not a derivation that reduces to its inputs.

full rationale

The paper contains no mathematical derivation, fitted parameters, or self-citation chain that would allow any claimed result to reduce to its own inputs. RoboPro is trained by supervised fine-tuning on 115k synthetic image-instruction-code triples produced by Video2Code from DROID videos, then evaluated on unseen RLBench, LIBERO, and real-world tasks. Section IV-A explicitly states that 'scenes, tasks and instructions during testing were entirely unseen during the training phase,' and the headline comparisons in Tables I, II, and IV are external success-rate measurements against CaP, GPT-4o, OpenVLA, and PerAct under the same API library. The unverified nature of the Video2Code synthetic code is a data-quality and attribution concern, not a circularity: the claim that performance stems from procedural knowledge learned from video is an empirical interpretation, and the shared API library is a deliberate experimental control rather than a definitional equivalence. No uniqueness theorem is imported, no load-bearing self-citation appears, and no existing result is renamed as a new contribution. The ablation comparing training with and without Video2Code may conflate learning API syntax with learning procedural ordering, but that is a confounded comparison, not a circular derivation. Thus the central claim is self-contained with respect to its evaluation and does not exhibit any of the enumerated circular patterns.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

There are no numeric free parameters fitted to data, but the central claim rests on several unverified domain assumptions: video-derived synthetic code quality, API sufficiency, and transferability of the training data. These are design choices and assumptions rather than fitted constants.

assumptions (4)
  • domain assumption DROID robot videos contain sufficient procedural knowledge to synthesize useful policy code for manipulation tasks.
    The entire Video2Code pipeline is built on this premise; if the videos lack the required task structure, the synthetic code will be low quality (Section III-B).
  • domain assumption Off-the-shelf Gemini-1.5-Flash and DeepSeek-Coder-V2 generate correct, executable code from video-derived plans without any verification.
    The paper does not evaluate the quality or executability of the 115k synthetic samples; this assumption is load-bearing for the training signal (Section III-B).
  • ad hoc to paper The hand-designed API library, including perception and control modules, is sufficient for all evaluation tasks and provides a fair interface across baselines.
    The API set is designed by the authors and shared across methods; if it is biased toward RoboPro's training distribution, the comparison may be unfair (Section IV and Appendix V-E).
  • domain assumption CodeQwen-1.5-7B and SigLIP-L are appropriate base models and that training on synthetic code plus general VLM data is sufficient to induce visual grounding.
    The architecture and training recipe are standard but not derived from first principles; the ablation shows only that a stronger code LLM helps (Section III-C).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation." pith.science (2026). https://pith.science/paper/IWEJWISG

@misc{pith2026250104268,
  author       = {Pith},
  title        = {Pith review of: Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IWEJWISG}},
  note         = {Machine review of arXiv:2501.04268}
}
read the original abstract

Zero-shot generalization across various robots, tasks and environments remains a significant challenge in robotic manipulation. Policy code generation methods use executable code to connect high-level task descriptions and low-level action sequences, leveraging the generalization capabilities of large language models and atomic skill libraries. In this work, we propose Robotic Programmer (RoboPro), a robotic foundation model, enabling the capability of perceiving visual information and following free-form instructions to perform robotic manipulation with policy code in a zero-shot manner. To address low efficiency and high cost in collecting runtime code data for robotic tasks, we devise Video2Code to synthesize executable code from extensive videos in-the-wild with off-the-shelf vision-language model and code-domain large language model. Extensive experiments show that RoboPro achieves the state-of-the-art zero-shot performance on robotic manipulation in both simulators and real-world environments. Specifically, the zero-shot success rate of RoboPro on RLBench surpasses the state-of-the-art model GPT-4o by 11.6%, which is even comparable to a strong supervised training baseline. Furthermore, RoboPro is robust to variations on API formats and skill sets.

Figures

Figures reproduced from arXiv: 2501.04268 by the authors.

Figure 1
Figure 1. Visualization of evaluation tasks and execution results. RoboPro shows impressive zero-shot performance on novel and compositional tasks in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The data curation pipeline of Video2Code. We first use the Draft VLM to extract a brief natural language plan for execution of the user instruction. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The overview of RoboPro. RoboPro utilizes environmental observation and natural language instruction as multimodal input, then outputs executable [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Error breakdown on RLBench. 2) LIBERO: We choose 8 representative tasks from LIBERO [32] as the evaluation set. These tasks include short￾horizon tasks which need scene understanding, and long￾horizon tasks which require multi-step implementation. Sim￾ilar with RLBench…
Figure 5
Figure 5. Figure 5: Success rate on manipulation tasks across varying data proportions. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Illustration of the selected tasks from LIBERO benchmark. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: The setup for real-world experiments. in the first two rows, the results on tow simulation platforms improved significantly after trained with Video2Code run￾time data, which indicates the effectiveness of 115k visual￾aligned code data collected from video demonstratio…
Figure 8
Figure 8. Figure 8: Illustration of RoboPro on the real-world experiments. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills

    cs.RO 2026-08 conditional novelty 6.0 of 10

    A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.

Reference graph

Works this paper leans on

35 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, Goyal, et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    arXiv 2023

  3. [3]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, Choromanski, et al. , “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818 , 2023

    arXiv 2023

  4. [4]

    Open x-embodiment: Robotic learning datasets and rt-x models,

    A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, Khazatsky, et al., “Open x-embodiment: Robotic learning datasets and rt-x models,” arXiv preprint arXiv:2310.08864 , 2023

    arXiv 2023

  5. [5]

    Openvla: An open-source vision-language- action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, Rafailov, et al. , “Openvla: An open-source vision-language- action model,” arXiv preprint arXiv:2406.09246 , 2024

    arXiv 2024

  6. [6]

    Robocodex: Multimodal code generation for robotic behavior synthesis,

    Y . Mu, J. Chen, Q. Zhang, S. Chen, Q. Yu, C. Ge, R. Chen, Z. Liang, M. Hu, C. Tao, et al. , “Robocodex: Multimodal code generation for robotic behavior synthesis,” arXiv preprint arXiv:2402.16117 , 2024

    arXiv 2024

  7. [7]

    Affordances from human videos as a versatile representation for robotics,

    S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak, “Affordances from human videos as a versatile representation for robotics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 778–13 790

    2023

  8. [8]

    Droid: A large-scale in-the-wild robot manipulation dataset,

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karam- cheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al., “Droid: A large-scale in-the-wild robot manipulation dataset,” arXiv preprint arXiv:2403.12945, 2024

    arXiv 2024

Show all 35 references
  1. [9]

    Perceiver-actor: A multi- task transformer for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi- task transformer for robotic manipulation,” in Conference on Robot Learning. PMLR, 2023, pp. 785–799

    2023

  2. [10]

    Procedures as a representation for data in a computer program for understanding natural language,

    T. Winograd, “Procedures as a representation for data in a computer program for understanding natural language,” 1971

    1971

  3. [11]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  4. [12]

    SAM-E: leveraging visual foundation model with sequence imitation for embodied manipulation,

    J. Zhang, C. Bai, H. He, Z. Wang, B. Zhao, X. Li, and X. Li, “SAM-E: leveraging visual foundation model with sequence imitation for embodied manipulation,” in International Conference on Machine Learning, 2024

    2024

  5. [13]

    Llarva: Vision-action instruction tuning enhances robot learning,

    D. Niu, Y . Sharma, G. Biamby, J. Quenum, Y . Bai, B. Shi, T. Darrell, and R. Herzig, “Llarva: Vision-action instruction tuning enhances robot learning,” arXiv preprint arXiv:2406.11815 , 2024

    2024 arXiv

  6. [14]

    Do as i can, not as i say: Grounding language in robotic affordances,

    A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al., “Do as i can, not as i say: Grounding language in robotic affordances,” in Conference on robot learning. PMLR, 2023, pp. 287–318

    2023

  7. [15]

    Palm-e: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, Wahid, et al., “Palm-e: An embodied multimodal language model,” in International Conference on Machine Learning . PMLR, 2023, pp. 8469–8488

    2023

  8. [16]

    Inner monologue: Embodied reasoning through planning with language models,

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar, et al. , “Inner monologue: Embodied reasoning through planning with language models,” in Conference on Robot Learning . PMLR, 2023, pp. 1769–1782

    2023

  9. [17]

    Progprompt: Generating situated robot task plans using large language models,

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 11 523–11 530

    2023

  10. [18]

    Code as policies: Language model programs for em- bodied control,

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for em- bodied control,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 9493–9500

    2023

  11. [19]

    Instruct2act: Mapping multi-modality instructions to robotic actions with large language model,

    S. Huang, Z. Jiang, H. Dong, Y . Qiao, P. Gao, and H. Li, “Instruct2act: Mapping multi-modality instructions to robotic actions with large language model,” arXiv preprint arXiv:2305.11176 , 2023

    2023 arXiv

  12. [20]

    V oxposer: Composable 3d value maps for robotic manipulation with language models,

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxposer: Composable 3d value maps for robotic manipulation with language models,” in Conference on Robot Learning . PMLR, 2023, pp. 540– 562

    2023

  13. [21]

    A unified approach for motion and force control of robot manipulators: The operational space formulation,

    O. Khatib, “A unified approach for motion and force control of robot manipulators: The operational space formulation,” IEEE Journal on Robotics and Automation , vol. 3, no. 1, pp. 43–53, 1987

    1987

  14. [22]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

    2023 arXiv

  15. [23]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,

    Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y . Wu, Li, et al., “Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,” arXiv preprint arXiv:2406.11931 , 2024

    2024 arXiv

  16. [24]

    Grounding dino: Marrying dino with grounded pre-training for open- set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, Yang, et al., “Grounding dino: Marrying dino with grounded pre-training for open- set object detection,” arXiv preprint arXiv:2303.05499 , 2023

    2023 arXiv

  17. [25]

    Anygrasp: Robust and efficient grasp perception in spatial and temporal domains,

    H.-S. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y . Xie, and C. Lu, “Anygrasp: Robust and efficient grasp perception in spatial and temporal domains,” IEEE Transactions on Robotics , 2023

    2023

  18. [26]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 11 975– 11 986

    2023

  19. [27]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, et al. , “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  20. [28]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26 296–26 306

    2024

  21. [29]

    Sharegpt4v: Improving large multi-modal models with better captions,

    L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin, “Sharegpt4v: Improving large multi-modal models with better captions,” arXiv preprint arXiv:2311.12793 , 2023

    2023 arXiv

  22. [30]

    Hello gpt-4o,

    OpenAI, “Hello gpt-4o,” May 2024. [Online]. Available: https: //openai.com/index/hello-gpt-4o/

    2024

  23. [31]

    Rlbench: The robot learning benchmark & learning environment,

    S. James, Z. Ma, D. Rovick Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,” IEEE Robotics and Automation Letters , 2020

    2020

  24. [32]

    Libero: Benchmarking knowledge transfer for lifelong robot learning,

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone, “Libero: Benchmarking knowledge transfer for lifelong robot learning,” Ad- vances in Neural Information Processing Systems , vol. 36, 2024

    2024

  25. [33]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, Bi, et al. , “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  26. [34]

    V-rep: A versatile and scalable robot simulation framework,

    E. Rohmer, S. P. Singh, and M. Freese, “V-rep: A versatile and scalable robot simulation framework,” in 2013 IEEE/RSJ international conference on intelligent robots and systems . IEEE, 2013, pp. 1321– 1326

    2013

  27. [35]

    Pyrep: Bringing v-rep to deep robot learning,

    S. James, M. Freese, and A. J. Davison, “Pyrep: Bringing v-rep to deep robot learning,” arXiv preprint arXiv:1906.11176 , 2019. APPENDIX A. Tasks in RLBench RLBench is a simulation platform set in CoppelaSim [34] and interfaced through PyRep [35]. Robotic models control a 7-do...

    1906 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.