Pith. sign in

REVIEW 6 major objections 5 minor 40 references

MA-ROESL: Motion-aware Rapid Reward Optimization for Efficient Robot Skill Learning from Single Videos

T0 review · 6 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Motion-aware frame selection and offline reward screening let a quadruped learn gaits from one video with 68.67 percent less training time.

desk verdict Plausible incremental pipeline for VLM-based skill learning, but the headline efficiency gain rests on an unspecified timing protocol and single-run numbers. read the letter →

arxiv 2505.08367 v1 pith:AKBSA2GZ submitted 2025-05-13 cs.RO

Xianghui Wang , Xinming Zhang , Yanjun Chen , Xiaoyu Shen , Wei Zhang This is my paper · ORCID
classification cs.RO
keywords robotskilllearningvision-languagemodelsrewarddesignofflinereinforcementsim-to-realtransferquadrupedlocomotionmotion-awareframeselectionsinglevideodemonstration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a quadruped robot can learn a locomotion skill from a single video demonstration with far less training time than existing methods, while still reproducing the demonstrated gait. Its proposal, MA-ROESL, changes the video-to-reward pipeline in two ways: it feeds a vision-language model frames selected by motion salience rather than uniform sampling, and it screens candidate rewards with offline reinforcement learning before a short online fine-tuning stage. On trot, pace, bound, and hop, the authors report an average 68.67 percent reduction in training time compared to the SDS baseline, with lower dynamic-time-warping mismatch to the demonstrations. The learned policies transfer zero-shot to a physical quadruped. If the claim holds, video-only skill acquisition becomes a practical route for legged locomotion rather than a costly research exercise.

What carries the argument

The load-bearing object is a motion-aware frame-selection score, the average dense optical-flow magnitude between consecutive frames, $$\sigma_k = \frac{1}{W H}\sum_{i,j} \|d_k(i,j)\|_2,$$ which picks frames that capture the stance-swing transitions that uniform sampling misses. The second mechanism is reward relabeling in an offline dataset: phase-1 trajectories are relabeled with each candidate reward, an offline policy is trained per relabeled dataset, and the vision-language model evaluates the resulting rollouts. This replaces several full online training runs with one offline screening pass before the final online fine-tuning stage.

What would settle it

Run the same set of candidate reward functions through both the phase-2 offline relabeling procedure and full online training, then evaluate every candidate online; if the offline-selected reward is consistently outperformed by an online-selected reward, the central efficiency claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in learning skills from a single video is not the vision-language model's ability to propose useful rewards, but how frames are sampled and how expensive reward evaluation is. MA-ROESL claims that a motion-aware frame selection step, based on dense optical flow, yields reward functions that better reflect stance-swing transitions, and that a three-phase pipeline—online data collection, offline reward relabeling with Implicit Q-Learning for cheap screening, then online fine-tuning—lets the method identify the best reward faster. The supporting evidence is the training-time table (16.83 to 6.15 hours for trot, 12.20 to 3.64 for pace, 13.10 to 4.16 for bound, 15.83 to 4.21 for hop), lower DTW alignment than SDS, matching contact patterns across the four gaits, and deployment of the policies on a physical quadruped.

Load-bearing premise

The offline dataset gathered in phase 1 must be rich enough that relabeling it with a candidate reward ranks candidates in the same order as full online training would; if that ranking is wrong, the accelerated screening selects a suboptimal reward and the efficiency gain is an artifact of the protocol.

Editorial extensions

If this is right

  • A single video can serve as the only demonstration source for learning a new quadruped gait, with training time short enough for iterative reward experimentation.
  • The 68.67 percent reduction makes it practical to compare many vision-language-proposed rewards in simulation before committing to online training, so reward search can be broader at the same compute budget.
  • Motion-aware frame selection is task-agnostic within the pipeline, so any vision-language-based reward generator can ingest better input frames without extra annotation.
  • Successful zero-shot deployment on a physical quadruped suggests policies trained this way preserve the dynamics of the demonstrated skill, not just its appearance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cheap test implied by the paper is to take one candidate reward and compare its offline relabeling ranking against its full online fitness; the efficiency argument predicts they agree, and a reader can check this directly.
  • Motion-aware sampling likely matters most for gaits with distinct stance and swing phases; for smooth, continuous motions the gain could shrink, so the four-skill result invites testing on non-periodic or long-horizon videos.
  • If offline reward screening were replaced by an even cheaper learned critic, the three-phase pipeline could be compressed further; the paper does not explore that route.
  • The reported times are per-skill on a single GPU workstation, so replication across other simulators and robot platforms would determine how much of the gain comes from offline screening speed versus the motion-aware frames.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes MA-ROESL, a framework for learning quadruped locomotion skills from a single video using vision-language models (VLMs). It introduces motion-aware frame selection based on optical flow to choose behaviorally salient frames for the VLM, and a three-phase training pipeline: (1) train PPO policies for VLM-generated candidate rewards and collect an offline dataset; (2) use IQL on relabeled rewards to rapidly select the best reward-policy pair; (3) fine-tune the selected policy online. The authors report a 68.67% average reduction in training time relative to the SDS baseline across trot, pace, bound, and hop, lower DTW motion-alignment values, and successful zero-shot sim-to-real deployment on a Unitree Go2.

Significance. If the reported efficiency gains are real, the paper addresses an important practical bottleneck in video-based reward learning for legged robots: the cost of evaluating many candidate rewards by full online training. The pipeline idea of using an offline dataset and IQL to rank rewards before online fine-tuning is timely and plausible, and the paper provides a clear problem formulation, explicit pseudocode, and a real-robot deployment, which are strengths. However, the central efficiency claim rests on an incompletely specified timing protocol and on single-number comparisons without statistical support, and the skill-quality evidence is largely qualitative. The significance is therefore conditional: the framework is potentially useful, but the current evidence does not substantiate the headline numbers.

major comments (6)
  1. [Algorithm 1, Section V-C.1, Table II] The timing protocol behind t_MA-ROESL is not defined, so the 68.67% reduction is not verifiable. Algorithm 1 phase 1 runs TrainAndCollect(M, R_i) for K1 reward functions per iteration for N1 iterations, i.e., K1*N1 full PPO trainings, each with 4000 environments; phase 2 runs K2*N2 IQL trainings; phase 3 runs an additional online fine-tune. The SDS baseline presumably also trains full PPO policies for each candidate reward. Unless phase 1 uses far fewer policy updates than the SDS runs, or phases are excluded from t_MA-ROESL, the comparison is not apples-to-apples. The paper never states which components are included in t_MA-ROESL, nor the values of K1, K2, N1, N2 or the number of PPO steps per policy. Please provide a per-phase timing breakdown and all protocol hyperparameters.
  2. [Table II, Section V-C.1] Table II reports a single training time per skill and per method, with no standard deviation, no number of seeds, and no statistical test. The claimed reduction of 63--73% is therefore a point estimate whose reliability is unknown. Since the main contribution is an efficiency claim, the evaluation needs multiple runs (at least 3--5 seeds) for both t_MA-ROESL and t_SDS, with the mean and spread reported.
  3. [Section V-A, Section V-C.1] The SDS baseline is not described: its environment code, reward prompt, frame sampling method, training algorithm settings, and hardware are not given. The paper only says SDS is the state-of-the-art method of [7]. Without a description of the baseline configuration, the reader cannot tell whether the time reduction reflects the proposed method or differences in implementation (e.g., number of reward candidates, PPO steps, or hardware). Please specify the exact SDS protocol used for the numbers in Table II.
  4. [Section IV-B.2, Algorithm 1 lines 13 and 24] The reward selection loop is self-referential: GPT-4V both generates candidate rewards and evaluates rollouts to select the best reward, in both phase 1 and phase 2. Since the same VLM is judge and generator, the selected reward may simply reflect the VLM's prior preference rather than actual task fitness F under the true MDP. The paper should discuss this circularity and provide a control experiment, e.g., comparing VLM-selected rewards against an independent fitness metric (such as task success rate or gait metrics) or against random selection.
  5. [Section IV-B.2, Eqs. (3)--(4)] The offline ranking assumption is load-bearing: phase 2 assumes that IQL policies trained on relabeled offline dataset D rank reward functions in the same order as their true fitness F after online training. The paper does not justify or test this assumption. A direct test would be to take the rewards selected by phase 2, train them fully online, and compare their final performance against rewards that phase 2 ranked lower; without such evidence, the efficiency gain could be an artifact of the offline ranking being wrong, with the final fine-tune masking the error.
  6. [Section V-C.2, Figs. 4--7] The skill-quality evaluation is qualitative: Fig. 4 shows DTW curves but no numerical DTW values or confidence intervals, and the claim that MA-ROESL has 'a lower DTW value' is not backed by numbers in the text or table. Similarly, Figs. 5--7 show contact patterns and joint angles but no quantitative success metric (e.g., percentage of steps with correct footfall sequence, cycle frequency error, or velocity tracking error). Please add quantitative metrics with uncertainty for both the simulation and real-world deployment.
minor comments (5)
  1. [Section III, Eq. (3)] Equation (3) is written as 'f = F Mun(π)' with inconsistent notation: the function name F appears on both sides, and the meaning of 'Mun' as the unknown MDP is not typeset clearly. Please rewrite to make the mapping explicit, e.g., f(π) = F_{M_un}(π).
  2. [Section IV-A, Eq. (6)] Equation (6) defines K_motion as a set of frame indices but the text below says 'K is the target number of motion-salient frames to be selected'; the paper should clarify whether K is a count or a rank threshold, and how the supplementary uniformly sampled frames are combined with the top-K selection.
  3. [Section V-B, hyperparameters] Table I lists PPO and IQL hyperparameters but omits the new method-specific parameters K, K1, K2, N1, N2, and the offline dataset size. These are needed to reproduce the experiments and to interpret the timing numbers.
  4. [Section V-C.2] The sentence about the amplitude discrepancy in thigh joints attributes it to 'the limitation of the input' but does not specify which input (the video frame selection, the VLM prompt, or the optical flow) or how this limitation could be mitigated. Please clarify.
  5. [Section V-A] The hardware description says the on-board Jetson Orin NX operates 'independently without external computational support,' but it is unclear whether this refers to running the trained policy or to training; training is stated to run on the workstation. Please disambiguate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the reported efficiency and skill-fidelity results are empirical comparisons, not derivations that reduce to their inputs.

full rationale

The paper does not present a first-principles derivation whose conclusion is equivalent to its assumptions. The core contribution is an empirical three-phase pipeline (Algorithm 1) whose reported outcomes are measured training times (Table II) and measured motion alignment via DTW and contact-pattern analysis. No parameter is fitted to the target quantity and then renamed as a prediction; the 68.67% reduction is a direct arithmetic comparison of reported wall-clock times, not a quantity derived from fitted values. The same VLM (GPT-4V) is used both to generate candidate rewards and to rank rollouts, which makes the internal reward-selection loop self-referential and weakens the claim that R_bst maximizes the unknown fitness F in Eq. (4), but this is an evaluation-proxy validity concern rather than a circularity in the sense of the derivation reducing to its inputs by construction: the paper also checks the final policies against video-derived DTW and contact-pattern signals, which are external to the VLM's reward generator. The absence of per-phase timing breakdowns and of values for K1, K2, N1, N2 is a reproducibility and comparability weakness in the efficiency claim, not a circular step. Citations to the authors' prior work ([11], [13]) appear in related-work and background contexts and are not load-bearing for the main claims. No load-bearing self-citation, uniqueness import, renamed known result, or fitted-input-called-prediction step can be exhibited under the required evidentiary standard, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. Its load-bearing assumptions are the domain assumptions listed above: optical-flow salience, offline ranking reliability, and VLM-as-judge fidelity. The central efficiency claim depends on iteration counts and dataset sizes that are not reported.

free parameters (4)
  • K (target number of motion-salient frames)
    Used in Eq. (6); its value is not listed in Table I, yet it directly controls which frames are sent to the VLM and thus shapes the generated rewards.
  • N1, N2 (phase 1 and 2 iteration counts)
    From Algorithm 1; these iteration counts directly determine training time, so the reported time reduction is sensitive to them, but their values are not given in the paper's hyperparameter table.
  • K1, K2 (number of reward functions per phase)
    From Algorithm 1; number of candidate rewards generated per iteration, which sets the cost of phase 1 and phase 2 and is not reported in Table I.
  • Offline dataset size
    The size of D collected in phase 1 determines whether IQL can reliably rank rewards; the paper does not report the number of transitions or policies used.
assumptions (4)
  • domain assumption Optical flow magnitude identifies behaviorally salient frames that improve VLM reward generation
    Used in Section IV-A; the paper provides no ablation or quantitative evidence that motion-salient frames materially change the VLM's reward quality.
  • domain assumption The offline dataset D collected from phase-1 policies is sufficiently diverse to rank VLM-generated reward functions via offline IQL
    Assumed in Section IV-B.2; if offline ranking disagrees with true fitness F, the efficiency claim collapses because the selected reward would not be the best.
  • domain assumption The VLM evaluation of rollouts is a faithful proxy for the true fitness F in Eq. (3)
    Both reward generation and reward selection use GPT-4V (Algorithm 1 lines 13 and 24); no calibration against a ground-truth reward is provided.
  • domain assumption Generated reward code is executable and compatible with the simulator API
    Phase 1 trains on VLM-generated reward code; the paper does not discuss how syntax or API errors are handled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MA-ROESL: Motion-aware Rapid Reward Optimization for Efficient Robot Skill Learning from Single Videos." pith.science (2026). https://pith.science/paper/AKBSA2GZ

@misc{pith2026250508367,
  author       = {Pith},
  title        = {Pith review of: MA-ROESL: Motion-aware Rapid Reward Optimization for Efficient Robot Skill Learning from Single Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKBSA2GZ}},
  note         = {Machine review of arXiv:2505.08367}
}
read the original abstract

Vision-language models (VLMs) have demonstrated excellent high-level planning capabilities, enabling locomotion skill learning from video demonstrations without the need for meticulous human-level reward design. However, the improper frame sampling method and low training efficiency of current methods remain a critical bottleneck, resulting in substantial computational overhead and time costs. To address this limitation, we propose Motion-aware Rapid Reward Optimization for Efficient Robot Skill Learning from Single Videos (MA-ROESL). MA-ROESL integrates a motion-aware frame selection method to implicitly enhance the quality of VLM-generated reward functions. It further employs a hybrid three-phase training pipeline that improves training efficiency via rapid reward optimization and derives the final policy through online fine-tuning. Experimental results demonstrate that MA-ROESL significantly enhances training efficiency while faithfully reproducing locomotion skills in both simulated and real-world settings, thereby underscoring its potential as a robust and scalable framework for efficient robot locomotion skill learning from video demonstrations.

Figures

Figures reproduced from arXiv: 2505.08367 by the authors.

Figure 1
Figure 1. Illustration of the skill learning problem from video [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The comparison of frame selection results: a) uniform [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of MA-ROESL framework. 1) Phase 1: VLM-generated reward functions are used to train policies and collect [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Evaluation of motion pattern alignment using dynamic [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Contact patterns of the four limbs across different [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Simulation and real-world evaluations using MA-ROESL-trained policies for four locomotion skills: a) Hop, b) Trot, c) [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Illustration of thigh joint positions generated and [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 15 canonical work pages

  1. [7]

    Sds–see it, do it, sorted: Quadruped skill synthesis from single video demonstration,

    J. Li, M. Stamatopoulou, and D. Kanoulas, “Sds–see it, do it, sorted: Quadruped skill synthesis from single video demonstration,”arXiv preprint arXiv:2410.11571, 2024

    arXiv 2024

  2. [1]

    Learning to walk in minutes using massively parallel deep reinforcement learning,

    N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Conference on Robot Learning. PMLR, 2022, pp. 91–100

    2022

  3. [2]

    Advanced skills through multiple adversarial motion priors in reinforcement learning,

    E. V ollenweider, M. Bjelonic, V . Klemm, N. Rudin, J. Lee, and M. Hutter, “Advanced skills through multiple adversarial motion priors in reinforcement learning,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 5120–5126

    2023

  4. [3]

    Walk these ways: Tuning robot control for generalization with multiplicity of behavior,

    G. B. Margolis and P. Agrawal, “Walk these ways: Tuning robot control for generalization with multiplicity of behavior,” inConference on Robot Learning. PMLR, 2023, pp. 22–31

    2023

  5. [4]

    Adversarial motion priors make good substitutes for complex reward functions,

    A. Escontrela, X. B. Peng, W. Yu, T. Zhang, A. Iscen, K. Goldberg, and P. Abbeel, “Adversarial motion priors make good substitutes for complex reward functions,” in2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 25–32

    2022

  6. [5]

    Learning agile skills via adversarial imitation of rough partial demonstrations,

    C. Li, M. Vlastelica, S. Blaes, J. Frey, F. Grimminger, and G. Mar- tius, “Learning agile skills via adversarial imitation of rough partial demonstrations,” inConference on Robot Learning. PMLR, 2023, pp. 342–352

    2023

  7. [6]

    Roboclip: One demonstration is enough to learn robot policies,

    S. Sontakke, J. Zhang, S. Arnold, K. Pertsch, E. Bıyık, D. Sadigh, C. Finn, and L. Itti, “Roboclip: One demonstration is enough to learn robot policies,”Advances in Neural Information Processing Systems, vol. 36, pp. 55 681–55 693, 2023

    2023

  8. [8]

    Mocapact: A multi-task dataset for simulated humanoid control,

    N. Wagener, A. Kolobov, F. Vieira Frujeri, R. Loynd, C.-A. Cheng, and M. Hausknecht, “Mocapact: A multi-task dataset for simulated humanoid control,”Advances in Neural Information Processing Systems, vol. 35, pp. 35 418–35 431, 2022

    work page 2022

Show all 40 references
  1. [9]

    Sfv: Re- inforcement learning of physical skills from videos,

    X. B. Peng, A. Kanazawa, J. Malik, P. Abbeel, and S. Levine, “Sfv: Re- inforcement learning of physical skills from videos,”ACM Transactions On Graphics (TOG), vol. 37, no. 6, pp. 1–14, 2018

    2018

  2. [10]

    Deep reinforcement learning-based safe interaction for industrial human-robot collaboration using intrinsic reward function,

    Q. Liu, Z. Liu, B. Xiong, W. Xu, and Y . Liu, “Deep reinforcement learning-based safe interaction for industrial human-robot collaboration using intrinsic reward function,”Advanced Engineering Informatics, vol. 49, p. 101360, 2021

    2021

  3. [11]

    Achieving stable high-speed locomotion for humanoid robots with deep reinforcement learning,

    X. Zhang, X. Wang, L. Zhang, G. Guo, X. Shen, and W. Zhang, “Achieving stable high-speed locomotion for humanoid robots with deep reinforcement learning,”arXiv preprint arXiv:2409.16611, 2024

    2024 arXiv

  4. [12]

    Mastering the game of go without human knowledge,

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton,et al., “Mastering the game of go without human knowledge,”nature, vol. 550, no. 7676, pp. 354–359, 2017

    2017

  5. [13]

    Drl- dclp: A deep reinforcement learning-based dimension-configurable local planner for robot navigation,

    W. Zhang, S. Wang, M. Tan, Z. Yang, X. Wang, and X. Shen, “Drl- dclp: A deep reinforcement learning-based dimension-configurable local planner for robot navigation,”IEEE Robotics and Automation Letters, 2025

    2025

  6. [14]

    Deep reinforcement learning for general game playing,

    A. Goldwaser and M. Thielscher, “Deep reinforcement learning for general game playing,” inProceedings of the AAAI conference on artificial intelligence, vol. 34, no. 02, 2020, pp. 1701–1708

    2020

  7. [15]

    Novel automated interactive reinforcement learning framework with a constraint-based supervisor for procedural tasks,

    ´I. Elguea-Aguinaco, A. Aguirre-Ortuzar, U. Izagirre-Aizpitarte, I. Inziarte-Hidalgo, S. Bøgh, and N. Arana-Arexolaleiba, “Novel automated interactive reinforcement learning framework with a constraint-based supervisor for procedural tasks,”Knowledge-Based Systems, vol. 309, p...

    2025

  8. [16]

    Deep reinforcement learning for unsu- pervised video summarization with diversity-representativeness reward,

    K. Zhou, Y . Qiao, and T. Xiang, “Deep reinforcement learning for unsu- pervised video summarization with diversity-representativeness reward,” inProceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018

    2018

  9. [17]

    Imitation from observation: Learning to imitate behaviors from raw video via context translation,

    Y . Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observation: Learning to imitate behaviors from raw video via context translation,” in 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 1118–1125

    2018

  10. [18]

    Reinforcement learning with videos: Combining offline observations with interaction,

    K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn, “Reinforcement learning with videos: Combining offline observations with interaction,”arXiv preprint arXiv:2011.06507, 2020

    2011 arXiv

  11. [19]

    Learning agile robotic locomotion skills by imitating animals,

    X. B. Peng, E. Coumans, T. Zhang, T. E. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” in Robotics: Science and Systems XVI, Virtual Event / Corvalis, Oregon, USA, July 12-16, 2020, M. Toussaint, A. Bicchi, and T. Hermans, Eds.,

    2020

  12. [20]

    Progprompt: Generating situated robot task plans using large language models,

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 11 523–11 530

    2023

  13. [21]

    Language to rewards for robotic skill synthesis,

    W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.- T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik,et al., “Language to rewards for robotic skill synthesis,”arXiv preprint arXiv:2306.08647, 2023

    2023 arXiv

  14. [22]

    Eureka: Human-level reward design via coding large language models,

    Y . J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y . Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https:/...

    2024

  15. [23]

    Dreureka: Language model guided sim-to-real transfer,

    Y . J. Ma, W. Liang, H. Wang, S. Wang, Y . Zhu, L. Fan, O. Bastani, and D. Jayaraman, “Dreureka: Language model guided sim-to-real transfer,” inRobotics: Science and Systems (RSS), 2024

    2024

  16. [24]

    Vision-language models are zero-shot reward models for reinforcement learning,

    J. Rocamonde, V . Montesinos, E. Nava, E. Perez, and D. Lindner, “Vision-language models are zero-shot reward models for reinforcement learning,”arXiv preprint arXiv:2310.12921, 2023

    2023 arXiv

  17. [25]

    Slomo: A general system for legged robot motion imitation from casual videos,

    J. Z. Zhang, S. Yang, G. Yang, A. L. Bishop, S. Gurumurthy, D. Ra- manan, and Z. Manchester, “Slomo: A general system for legged robot motion imitation from casual videos,”IEEE Robotics and Automation Letters, vol. 8, no. 11, pp. 7154–7161, 2023

    2023

  18. [26]

    Continuous control with deep reinforcement learning,

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Tra...

    2016 arXiv

  19. [27]

    A survey on offline reinforcement learning: Taxonomy, review, and open problems,

    R. F. Prudencio, M. R. Maximo, and E. L. Colombini, “A survey on offline reinforcement learning: Taxonomy, review, and open problems,” IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  20. [28]

    Deep reinforcement learning for autonomous driving: A survey,

    B. R. Kiran, I. Sobh, V . Talpaert, P. Mannion, A. A. Al Sallab, S. Yo- gamani, and P. P ´erez, “Deep reinforcement learning for autonomous driving: A survey,”IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 4909–4926, 2021

    2021

  21. [29]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

    S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,”arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  22. [30]

    Off-policy deep reinforcement learning without exploration,

    S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” inProceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09...

    2019

  23. [31]

    Conservative q-learning for offline reinforcement learning,

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,”Advances in neural information processing systems, vol. 33, pp. 1179–1191, 2020

    2020

  24. [32]

    Where do rewards come from,

    S. Singh, R. L. Lewis, and A. G. Barto, “Where do rewards come from,” inProceedings of the annual conference of the cognitive science society. Cognitive Science Society, 2009, pp. 2601–2606

    2009

  25. [33]

    Eureka: Human- level reward design via coding large language models,

    Y . J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Ja- yaraman, Y . Zhu, L. Fan, and A. Anandkumar, “Eureka: Human- level reward design via coding large language models,”arXiv preprint arXiv:2310.12931, 2023

    2023 arXiv

  26. [34]

    Two-frame motion estimation based on polynomial ex- pansion,

    G. Farneb ¨ack, “Two-frame motion estimation based on polynomial ex- pansion,” inImage Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29–July 2, 2003 Proceedings 13. Springer, 2003, pp. 363–370

    2003

  27. [35]

    Off-policy deep reinforcement learning without exploration,

    S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” inInternational conference on machine learning. PMLR, 2019, pp. 2052–2062

    2019

  28. [36]

    Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble,

    S. Lee, Y . Seo, K. Lee, P. Abbeel, and J. Shin, “Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble,” inConference on Robot Learning. PMLR, 2022, pp. 1702–1712

    2022

  29. [37]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  30. [38]

    Offline reinforcement learning with implicit q-learning,

    I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit q-learning,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https: //openreview.net/forum?id=68n2s9ZJWF8

    2022

  31. [39]

    Gpt-4v(ision) system card,

    “Gpt-4v(ision) system card,” 2023. [Online]. Available: https://api. semanticscholar.org/CorpusID:263218031

    2023

  32. [2020]

    Available: https://doi.org/10.15607/RSS.2020.XVI.064

    [Online]. Available: https://doi.org/10.15607/RSS.2020.XVI.064

    2020 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.