Pith. sign in

REVIEW 4 major objections 5 minor 54 references

This paper claims that dexterous manipulation skills can be encoded as a single linear dynamical system in a learned latent space, yielding an implicit full-horizon planner that predicts actions and visual flow together.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:36 UTC pith:LTTK5QFK

load-bearing objection Genuinely interesting Koopman-behavioral policy class with credible held-out sim results, but the full-horizon open-loop 'implicit planner' premise is under-supported by the training objective and the paper's own drift curves. the 4 major comments →

arxiv 2602.07413 v3 pith:LTTK5QFK submitted 2026-02-07 cs.RO

Going with the Flow: Koopman Behavioral Models as Pseudo Planners for Visuo-Motor Dexterity

Yunhai Han , Jiaqi Fu , Linhao Bai , Ziyu Xiao , Zhaodong Yang , Yogita Choudhary , Krishna Jha , Chuizheng Kong
show 2 more authors
Shreyas Kousik Harish Ravichandar
This is my paper
classification cs.RO
keywords Koopman operatorimitation learningdexterous manipulationvisuo-motor policylatent linear dynamicsreplanningvisual flow predictionrobotics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most visuo-motor dexterity models treat a skill as a reactive map from observation to action chunks, trading temporal coherence against reactivity. This paper argues instead that a dexterous skill can be captured as a single coupled dynamical system: a unified latent state in which visual features of the manipulated object and the robot's own actions co-evolve. It instantiates this idea as Koopman-UBM, a state-inclusive latent that evolves linearly under one learned matrix K, so a full-horizon plan (actions plus imagined visual flow) is computed analytically from an initial condition by repeated application of K. The authors report that this implicit planner matches or exceeds diffusion- and transformer-based baselines across dexterous tasks while running at about 0.42 ms per step, and that its own predicted-versus-observed visual divergence can trigger replanning. The payoff, if true, is a middle path between deliberate full-horizon planning and fast reactivity, with temporal coherence guaranteed by construction rather than by averaging.

Core claim

The central claim, stated on the paper's own terms, is that the joint flow of visual features and robot actions during a dexterous skill can be represented as a linear dynamical system in a learned lifted space. The paper proposes a 'state-inclusive' latent z_t = [ξ_t; ψ_t], where ξ_t is the current behavioral state (robot action plus compressed visual features) and ψ_t is an MLP lift, governed by z_{t+1} = K z_t. Trained with a multi-step coherence loss L = E_t Σ_l ||K^l z_t − z_{t+l}||², K-UBM acts as an implicit planner: from an initial condition it computes the desired robot behavior and simultaneously predicts the flow of visual features over the entire skill horizon. The same predicted

What carries the argument

The load-bearing mechanism is the Koopman-UBM latent linear system. A unified behavioral state ξ_t concatenates the robot action a_t with a compressed visual feature φ_t (either tracked object-flow points or self-supervised dynamics features). An MLP lifting function maps ξ_t to ψ_t, and the joint latent z_t = [ξ_t; ψ_t] is evolved by a learned matrix K. Because ξ_t is included verbatim in z_t, predicted actions and visual features can be read out without an extra decoder, and the same matrix is applied repeatedly to generate arbitrarily long rolling plans. Training minimizes the multi-step coherence loss over a horizon H rather than a one-step error, and two stabilizers — identity initializ

Load-bearing premise

The whole method rests on the premise that one fixed linear rule can accurately simulate the coupled hand-object-vision dynamics of a contact-rich skill from start to finish; if that rule drifts or misses sudden contact events, the open-loop plan degrades — and the paper itself concedes that linearization smooths over contact discontinuities.

What would settle it

In a contact-rich task such as hammering, compare predicted versus observed object-flow points at the instant of nail impact: if prediction error jumps sharply at impact across trials, the single linear operator misses contact discontinuities; if it stays flat, the linearization claim holds. The paper's own Fig. 6 already shows rising flow-prediction RMSE with trajectory percentile, which is the direction such a test would take.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Temporal coherence becomes a property of the representation, not a heuristic: the same K can be rolled out for 1 step or the whole skill, so planning horizon becomes a tunable knob instead of a fixed hyperparameter.
  • Because action and visual-flow predictions come from the same linear operator, the model can act as its own verifier: predicted object motion can flag failures before they complete and can serve as a reward signal for downstream learning.
  • Open-loop rollout gives resilience to visual occlusion or camera failure: the learned dynamics bridge gaps where reactive policies freeze.
  • Event-triggered replanning turns the coherence-versus-reactivity trade-off into a single threshold: execute while predictions match reality, replan only when they diverge.
  • Inference cost is dominated by matrix multiplication (reported around 0.42 ms per step), making the approach practical for real-time multi-fingered control.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the linearization holds across a task family, the spectral properties of K (eigenvalues, spectral radius) may become an analytic descriptor of a skill — a way to compare, interpolate, or certify skills before execution, which the paper does not explore.
  • The replanning threshold could be tuned statistically from the flow-prediction error distribution on successful rollouts, converting a hand-set threshold into a calibrated test; the paper leaves threshold selection unspecified.
  • The framework invites composition: if two skills share a latent subspace, their Koopman operators might be blended or switched to produce new skills, though the paper's own limitation about contact discontinuities suggests switching operators rather than a single K for high-frequency impacts.
  • The claimed occlusion robustness is a direct corollary of open-loop prediction, but the paper does not quantify it for K-UBM itself — its robustness appendix tests only reactive baselines — so measuring K-UBM under tracking failure is a natural follow-up.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Unified Behavioral Models (UBMs), a framework for representing dexterous manipulation skills as coupled dynamical systems over robot state and visual features, and instantiates it as Koopman-UBM (K-UBM). K-UBM learns a state-inclusive latent z_t = [ξ_t; ψ_t] with linear dynamics z_{t+1} = K z_t, trained by a multi-step coherence loss, and uses open-loop latent rollouts as an implicit planner. An event-triggered replanning scheme monitors predicted vs. observed visual features. The method is evaluated on seven simulated tasks and two real-world tasks, comparing against Diffusion Policy, ACT, KODex, and KOROL, and is reported to match or exceed baseline success rates with much lower inference cost and to support robustness and replanning.

Significance. If the claims hold, the paper offers a compelling alternative to reactive visuo-motor policies: a single learned linear operator that generates coherent full-horizon action and visual-feature plans, with fast inference and a built-in predictive replanning signal. The strengths of the manuscript include evaluation over seven simulation tasks with five seeds, careful baseline hyperparameter tuning reported in the appendices, real-world validation on two tasks, and an ablation study of the Koopman learning recipe. However, several load-bearing claims—notably full-horizon open-loop accuracy, occlusion robustness, and the reliability of the replanning monitor—are not yet adequately supported by the evidence presented. The core idea is promising and the paper is generally clear, but the missing analyses are central to the stated contributions.

major comments (4)
  1. [Sec. III.B and Eq. (3); Sec. X; Fig. 6] The implicit-planner claim requires that K^l z_t remains accurate over the full skill horizon, but the coherence loss is minimized only for l=1..H, with H=15 in simulation and H=50 in real-world tasks, while real-world trajectories exceed 150 steps. Fig. 6 shows flow-prediction RMSE increasing steadily across trajectory percentiles and DynaMo cosine similarity decreasing over time, which is exactly the regime where the open-loop planner operates. The Limitations (Sec. VI) further concede that linear dynamics smooth over contact discontinuities. To support the claim of full-horizon planning, the paper should report multi-step prediction error at horizons beyond H, evaluate open-loop success as a function of rollout length, and compare predicted-vs-observed divergence during nominal execution to quantify late-horizon drift. Without this, the replanning monitor's reliability at late times i
  2. [Abstract; Sec. I; Sec. VI; Appendix XIV] The paper repeatedly claims robustness to occlusions as a key advantage, but no experiment evaluates K-UBM under occlusion or corrupted visual input. Appendix XIV explicitly test only reactive policies (Diffusion, ACT) under blacked-out images and omits K-UBM. Since the robustness mechanism is open-loop execution that bridges perceptual gaps, a direct comparison is needed: run K-UBM with dropped frames, blacked-out images, or feature-tracking failures and report success rates. This is a load-bearing claim in the abstract and contribution list, not a secondary detail.
  3. [Abstract; Sec. IV.B; Sec. V; Table III] The stated number of real-world tasks is inconsistent. The abstract appearing before the full text says 'four real-world tasks,' and Sec. IV.B says 'four real-world tasks,' but Sec. V and Table III present only two real-world tasks (Uncover Pot and Open Lid). The hyperparameter table in Sec. X also says 'four real-world tasks.' This discrepancy must be corrected; if only two tasks were performed, all instances of 'four' should be changed, and the claims in the abstract and introduction should be aligned.
  4. [Sec. IV.D; Table II; Fig. 9] The event-triggered replanning is a central contribution, but the detection results are incomplete. For the flow-based policy, the paper reports 30/30 trigger detection, yet final success is only 15/30 for Door and 27/30 for Relocation, and the DynaMo-based trigger succeeds in only 3/30 and 21/30 cases. The paper does not report false-positive rates during unperturbed rollouts, trigger latency, or sensitivity to the detection threshold. Since Table II shows detection accuracy does not translate into task success, the paper should provide a more rigorous characterization of the replanning monitor, including ROC-like analysis over thresholds and a breakdown of failures after a correct trigger.
minor comments (5)
  1. [Table I] The inference-time comparison excludes visual feature extraction for all methods. For reactive policies, feature extraction is incurred at every control step, while K-UBM pays it once at initialization. The table should either include end-to-end per-step cost or explicitly state that feature extraction is excluded for all methods and explain why this does not bias the comparison.
  2. [Fig. 3 and Tables VI–IX] Success rates are reported with standard deviations over five seeds, but no statistical significance tests or confidence intervals are provided. Given the overlap in many cells, the claim that K-UBM 'matches or exceeds' baselines would be strengthened by pairwise tests or effect sizes.
  3. [Table II] The table reports success counts out of 30 but does not state whether these episodes come from a single policy seed or multiple seeds. For consistency with the rest of the evaluation, this should be clarified.
  4. [Appendix XIV, Fig. 10] The observation that the Door task improves under noise 'due to the policy exploiting unrealistic behaviors in simulation' is concerning. This should be explained or the result removed; otherwise it undermines the otherwise useful robustness comparison.
  5. [Sec. IV.C, Fig. 4 and Fig. 5] The qualitative visualizations of predicted flow points and DynaMo features are helpful, but no quantitative comparison against an alternative predictor is provided. Reporting the RMSE/cosine-similarity of, e.g., a constant-velocity baseline or a one-step linear predictor would contextualize the values in Fig. 6.

Circularity Check

0 steps flagged

No load-bearing circularity: K-UBM is a supervised fit evaluated on held-out test episodes; self-citations are baselines/design references, not the argument's substrate.

full rationale

K-UBM's derivations are not circular: the Koopman operator K and encoder gθ are fitted to demonstrations by minimizing the multi-step coherence loss L_K-coherence = E_t Σ_l ||K^l z_t − z_{t+l}||² (Sec. III.B), and the reported success rates, flow-prediction RMSE, and replanning results are measured on held-out test episodes against external baselines (Diffusion, ACT, KODex, KOROL). The 'implicit planner' language is an interpretation of open-loop rollout z_{t+l}=K^l z_t, not a claim that the rollout is correct by construction; the fitted dynamics are validated empirically. Self-citations such as KODex [41], KOROL [42], and design references [13,14,22,44,49] appear as baselines or implementation choices, not as the load-bearing justification for the central prediction claim; no uniqueness theorem from the authors is invoked to force the chosen model. The acknowledged limitations (Sec. VI) and the growing prediction error in Fig. 6 are accuracy/generalization concerns, not evidence that a prediction reduces to its training input by construction.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The paper's contribution is a learned representation plus fitted linear dynamics. Everything the claim rests on: (a) the Koopman matrix and lifting MLP fitted by the multi-step prediction loss; (b) hyperparameters (lift dim 256, H=15) chosen by experimentation; (c) the under-reported replanning threshold; (d) pretrained/self-supervised feature extractors (CoTracker+SAM3, DynaMo, flow autoencoder). No new physical entities are introduced.

free parameters (7)
  • Koopman matrix K = d_z × d_z with d_z = d_ξ + 256 (Table IV)
    The central fitted object; learned via the multi-step linear coherence loss (Eq. 1). All rollout 'predictions' are powers of K.
  • Lifting MLP θ (unified spectral encoder) = MLP hidden (128, 256), ReLU (Table IV)
    Learned jointly with K to make latent dynamics linear; determines what K can capture.
  • Multi-step prediction horizon H = 15 (sim), 50 (real-world; Sec. X)
    Chosen hyperparameter for the coherence loss; affects drift accumulation during training and rollout length.
  • Replanning trigger threshold = not reported
    The divergence threshold that fires replanning (Sec. IV.D) is never given a value or selection procedure; detection is reported 30/30 on the perturbation scenarios, but the threshold itself is unauditable.
  • Visual feature rescaling constant = computed from demo l2 norms (Sec. III.B)
    Normalizes visual features relative to actions in the unified state; a data-computed constant.
  • Flow autoencoder (128-dim) = 128-dim latent (2×8×8), conv AE (Appendix VIII)
    Trained separately on flow frames; defines the visual feature space K acts on.
  • Learning-rate ratio and identity init = encoder lr 5e-4, Koopman lr 5e-5, identity init (Table IV, App. XV)
    Training-stability recipe identified by experimentation; the ablation shows training is unstable without it.
axioms (5)
  • domain assumption Finite-dimensional Koopman linearization: a fixed K approximates the nonlinear coupled dynamics in the lifted space over the full skill horizon (z_{t+1} = K z_t, Sec. III.B).
    The entire implicit-planning mechanism requires this; the paper itself concedes contact discontinuities are smoothed over (Sec. VI) and drift grows with horizon (Fig. 6).
  • domain assumption The compressed visual features (128-dim flow or DynaMo embeddings) are a sufficient state for planning the skill.
    If tracking or self-supervised features lose task-relevant object state, the dynamics model is blind to it; occlusion robustness is claimed but not tested for K-UBM (Appendix XIV).
  • domain assumption Demonstrations from RL experts (30/50 per task, Sec. IV.A) cover the test distribution well enough for open-loop rollouts.
    The method is imitation learning; out-of-distribution behavior is not guaranteed, and the reactivity experiments had to introduce perturbations to probe it.
  • standard math State-inclusive lifting (ξ_t as a subvector of z_t) grounds the latent space and enables decoder-free inference (after Korda & Mezić [31]).
    Borrowed construction from [31]; adopted to avoid drift and remove the action decoder.
  • standard math Koopman operator spectral theory background (Refs. [24-28]).
    Unproved background: existence of Koopman eigenfunctions; the paper approximates a finite invariant subspace.

pith-pipeline@v1.3.0-alltime-deepseek · 20810 in / 18360 out tokens · 172072 ms · 2026-08-03T03:36:30.835197+00:00 · methodology

0 comments
read the original abstract

Contemporary visuo-motor dexterity models often rely on expressive policy classes with diffusion and transformer backbones to achieve strong performance. However, these architectures require significant data and computational resources, and remain far from reliable, particularly for multi-fingered dexterity. Importantly, they model skills as reactive mappings and rely on fixed-horizon action chunking, creating a rigid trade-off between temporal coherence and reactivity. To address these issues, we first introduce Unified Behavioral Models (UBMs), a framework to represent dexterous skills as coupled dynamical systems that capture how visual features of the environment (visual flow) and proprioceptive states of the robot (action flow) co-evolve. As such, UBMs ensure temporal coherence by construction rather than heuristic averaging. Unlike world models that attempt to predict the impact of arbitrary robot actions on the environment, UBMs target behavioral dynamics that encode how demonstrated robot behavior is related to desired impacts on the environment. A UBM can be viewed as a pseudo planner: given an initial condition, it computes the desired robot behavior over the entire skill horizon, while simultaneously ``imagining" the resulting flow of visual features. To operationalize UBMs, we propose Koopman-UBM, a first instantiation of UBMs as a structured latent linear system. K-UBM is computationally efficient, enabling reactivity and adaptation via an online replanning strategy: the model acts as its own runtime monitor, automatically triggering replanning when predicted and observed visual flow diverge beyond a threshold. Across seven simulated tasks and four real-world tasks, our approach matches or exceeds the performance of state-of-the-art baselines, while offering considerably faster inference, smooth execution, robustness to occlusions, and flexible replanning.

Figures

Figures reproduced from arXiv: 2602.07413 by Chuizheng Kong, Harish Ravichandar, Jiaqi Fu, Krishna Jha, Linhao Bai, Shreyas Kousik, Yogita Choudhary, Yunhai Han, Zhaodong Yang, Ziyu Xiao.

Figure 1
Figure 1. Figure 1: (A) Standard reactive policies (e.g., Diffusion, ACT) map observations to short-horizon action chunks, lacking a consistent internal model of the future or memory beyond the observation window, leading to temporal incoherence and hand-coded chunk lengths. (B) In contrast, Unified Behavioral Models (UBM) model skills as joint behavioral dynamics of the robot and environment governing a continuous flow in a … view at source ↗
Figure 2
Figure 2. Figure 2: We evaluate K-UBM on seven simulations tasks and two real-world tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: We report the task success rates for each method on the test sets using both visual features (Flow on the top row and DynaMo on [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: We visualize the predicted flow points by decoding the flow [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: We visualize the predicted DynaMo features by projecting [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: In the top row, we show the reconstruction error (RMSE) between the predicted and ground-truth flow points. In the bottom row, we [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: For both tasks, the left column shows the robot executing [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Left: we demonstrate the prompts used for SAM3 to generate [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: For both visual features, we demonstrate how replanning [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: For both visual features, we observe that performance of reactive policies often drops significantly under corrupted visual inputs, [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: We conduct ablation studies to evaluate two key factors that enable efficient Koopman learning. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 14 linked inside Pith

  1. [1]

    An overview of dexterous manipulation

    Allison M Okamura, Niels Smaby, and Mark R Cutkosky. An overview of dexterous manipulation. InProceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Pro- ceedings (Cat. No. 00CH37065), volume 1, pages 255– 262, 2000

    2000

  2. [2]

    Dexs- ingrasp: Learning a unified policy for dexterous object singulation and grasping in cluttered environments.arXiv preprint arXiv:2504.04516, 2025

    Lixin Xu, Zixuan Liu, Zhewei Gui, Jingxiang Guo, Zeyu Jiang, Zhixuan Xu, Chongkai Gao, and Lin Shao. Dexs- ingrasp: Learning a unified policy for dexterous object singulation and grasping in cluttered environments.arXiv preprint arXiv:2504.04516, 2025

    arXiv 2025

  3. [3]

    Dexterous manipulation through imitation learning: A survey.arXiv preprint arXiv:2504.03515, 2025

    Shan An, Ziyu Meng, Chao Tang, Yuning Zhou, Tengyu Liu, Fangqiang Ding, Shufang Zhang, Yao Mu, Ran Song, Wei Zhang, et al. Dexterous manipulation through imitation learning: A survey.arXiv preprint arXiv:2504.03515, 2025

    arXiv 2025

  4. [4]

    Dexumi: Using human hand as the universal manipulation in- terface for dexterous manipulation.arXiv preprint arXiv:2505.21864, 2025

    Mengda Xu, Han Zhang, Yifan Hou, Zhenjia Xu, Linxi Fan, Manuela Veloso, and Shuran Song. Dexumi: Using human hand as the universal manipulation in- terface for dexterous manipulation.arXiv preprint arXiv:2505.21864, 2025

    arXiv 2025

  5. [5]

    Rotating without seeing: Towards in-hand dexterity through touch.arXiv preprint arXiv:2303.10880, 2023

    Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, and Xiaolong Wang. Rotating without seeing: Towards in-hand dexterity through touch.arXiv preprint arXiv:2303.10880, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Dexteritygen: Foundation controller for un- precedented dexterity.arXiv preprint arXiv:2502.04307, 2025

    Zhao-Heng Yin, Changhao Wang, Luis Pineda, Fran- cois Hogan, Krishna Bodduluri, Akash Sharma, Patrick Lancaster, Ishita Prasad, Mrinal Kalakrishnan, Jitendra Malik, et al. Dexteritygen: Foundation controller for un- precedented dexterity.arXiv preprint arXiv:2502.04307, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Robopanoptes: The all-seeing robot with whole-body dexterity.arXiv preprint arXiv:2501.05420, 2025

    Xiaomeng Xu, Dominik Bauer, and Shuran Song. Robopanoptes: The all-seeing robot with whole-body dexterity.arXiv preprint arXiv:2501.05420, 2025

    arXiv 2025

  8. [8]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    2025

  9. [9]

    Goal-Conditioned Imitation Learning us- ing Score-based Diffusion Policies

    Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-Conditioned Imitation Learning us- ing Score-based Diffusion Policies. InProceedings of Robotics: Science and Systems, Daegu, Republic of Korea, 7 2023. doi: 10.15607/RSS.2023.XIX.028

    doi:10.15607/rss.2023.xix.028 2023

  10. [10]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.arXiv preprint arXiv:2403.03954, 2024

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.arXiv preprint arXiv:2403.03954, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Learning fine-grained bimanual manipulation with low-cost hardware.Robotics: Science and Systems XIX, 2023

    Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware.Robotics: Science and Systems XIX, 2023

    2023

  12. [12]

    Bidirectional decoding: Improving action chunking via closed-loop resampling.arXiv preprint arXiv:2408.17355, 2024

    Yuejiang Liu, Jubayer Ibn Hamid, Annie Xie, Yoonho Lee, Maximilian Du, and Chelsea Finn. Bidirectional decoding: Improving action chunking via closed-loop resampling.arXiv preprint arXiv:2408.17355, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Immimic: Cross-domain imitation from human videos via mapping and interpolation.arXiv preprint arXiv:2509.10952, 2025

    Yangcen Liu, Woo Chul Shin, Yunhai Han, Zhenyang Chen, Harish Ravichandar, and Danfei Xu. Immimic: Cross-domain imitation from human videos via mapping and interpolation.arXiv preprint arXiv:2509.10952, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Sail: Faster-than-demonstration execution of imitation learning policies.arXiv preprint arXiv:2506.11948, 2025

    Nadun Ranawaka Arachchige, Zhenyang Chen, Wonsuhk Jung, Woo Chul Shin, Rohan Bansal, Pierre Barroso, Yu Hang He, Yingyang Celine Lin, Benjamin Joffe, Shreyas Kousik, et al. Sail: Faster-than-demonstration execution of imitation learning policies.arXiv preprint arXiv:2506.11948, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Dynamical movement primitives: learning attractor models for motor behaviors

    Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2):328–373, 2013

    2013

  16. [16]

    Dynamic movement prim- itives in robotics: A tutorial survey.The International Journal of Robotics Research, 42(13):1133–1184, 2023

    Matteo Saveriano, Fares J Abu-Dakka, Alja ˇz Kram- berger, and Luka Peternel. Dynamic movement prim- itives in robotics: A tutorial survey.The International Journal of Robotics Research, 42(13):1133–1184, 2023

    2023

  17. [17]

    Recent advances in robot learning from demonstration.Annual review of control, robotics, and autonomous systems, 3:297–330, 2020

    Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration.Annual review of control, robotics, and autonomous systems, 3:297–330, 2020

    2020

  18. [18]

    Mohammad Khansari-Zadeh and Aude Billard

    S. Mohammad Khansari-Zadeh and Aude Billard. Learn- ing Stable Nonlinear Dynamical Systems With Gaussian Mixture Models.IEEE Transactions on Robotics, 27 (5):943–957, October 2011. ISSN 1941-0468. doi: 10.1109/TRO.2011.2159412. Conference Name: IEEE Transactions on Robotics

    arXiv 2011

  19. [19]

    Neural dynamic policies for end-to-end sensorimotor learning.Advances in Neural Information Processing Systems, 33:5058–5069, 2020

    Shikhar Bahl, Mustafa Mukadam, Abhinav Gupta, and Deepak Pathak. Neural dynamic policies for end-to-end sensorimotor learning.Advances in Neural Information Processing Systems, 33:5058–5069, 2020

    2020

  20. [20]

    Geometric fabrics: a safe guiding medium for policy learning

    Karl Van Wyk, Ankur Handa, Viktor Makoviychuk, Yijie Guo, Arthur Allshire, and Nathan D Ratliff. Geometric fabrics: a safe guiding medium for policy learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6537–6543. IEEE, 2024

    2024

  21. [21]

    Neural Geometric Fabrics: Efficiently Learning High- Dimensional Policies from Demonstration

    Mandy Xie, Karl Van Wyk, Ankur Handa, Stephen Tyree, Dieter Fox, Harish Ravichandar, and Nathan Ratliff. Neural Geometric Fabrics: Efficiently Learning High- Dimensional Policies from Demonstration. InConference on Robot Learning (CoRL), 2022

    2022

  22. [22]

    On the surprising effectiveness of spectrum clipping in learning stable linear dynamics.arXiv preprint arXiv:2412.01168, 2024

    Hanyao Guo, Yunhai Han, and Harish Ravichandar. On the surprising effectiveness of spectrum clipping in learning stable linear dynamics.arXiv preprint arXiv:2412.01168, 2024

    arXiv 2024

  23. [23]

    Elastic motion policy: An adaptive dy- namical system for robust and efficient one-shot imitation learning.arXiv preprint arXiv:2503.08029, 2025

    Tianyu Li, Sunan Sun, Shubhodeep Shiv Aditya, and Nadia Figueroa. Elastic motion policy: An adaptive dy- namical system for robust and efficient one-shot imitation learning.arXiv preprint arXiv:2503.08029, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    B. O. Koopman. Hamiltonian Systems and Transfor- mation in Hilbert Space.Proceedings of the National Academy of Sciences, 17(5):315–318, 1931. doi: 10. 1073/pnas.17.5.315

    1931

  25. [25]

    Springer, 2020

    Alexandre Mauroy, Y Susuki, and Igor Mezi ´c.Koopman operator in systems and control. Springer, 2020

    2020

  26. [26]

    Model-based control using koopman operators

    Ian Abraham, Gerardo De La Torre, and Todd D Mur- phey. Model-based control using koopman operators. arXiv preprint arXiv:1709.01568, 2017

    Pith/arXiv arXiv 2017

  27. [27]

    Koopman-based control of a soft continuum manipulator under variable loading conditions.IEEE robotics and automation letters, 6(4): 6852–6859, 2021

    Daniel Bruder, Xun Fu, R Brent Gillespie, C David Remy, and Ram Vasudevan. Koopman-based control of a soft continuum manipulator under variable loading conditions.IEEE robotics and automation letters, 6(4): 6852–6859, 2021

    2021

  28. [28]

    Koopman operators in robot learning.IEEE Transactions on Robotics, 2026

    Lu Shi, Masih Haseli, Giorgos Mamakoukas, Daniel Bruder, Ian Abraham, Todd Murphey, Jorge Cort ´es, and Konstantinos Karydis. Koopman operators in robot learning.IEEE Transactions on Robotics, 2026

    2026

  29. [29]

    Cotracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. InEuropean conference on computer vision, pages 18–35. Springer, 2024

    2024

  30. [30]

    Dynamo: In-domain dynamics pretraining for visuo-motor control.Advances in Neural Information Processing Systems, 37:33933–33961, 2024

    Zichen Cui, Hengkai Pan, Aadhithya Iyer, Siddhant Hal- dar, and Lerrel Pinto. Dynamo: In-domain dynamics pretraining for visuo-motor control.Advances in Neural Information Processing Systems, 37:33933–33961, 2024

    2024

  31. [31]

    Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control.Automatica, 93:149–160, July

    Milan Korda and Igor Mezi ´c. Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control.Automatica, 93:149–160, July

  32. [32]

    V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, June 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Gar- rido, Russell Howes, Mojtaba, Komeili, Matthew Muck- ley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Fran- cois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong L...

    Pith/arXiv arXiv 2025

  33. [33]

    Learning Latent Action World Models In The Wild, January 2026

    Quentin Garrido, Tushar Nagarajan, Basile Terver, Nico- las Ballas, Yann LeCun, and Michael Rabbat. Learning Latent Action World Models In The Wild, January 2026. URL http://arxiv.org/abs/2601.05230. arXiv:2601.05230 [cs]

    arXiv 2026

  34. [34]

    Christensen, Hao Su, Jiajun Wu, and Yunzhu Li

    Bo Ai, Stephen Tian, Haochen Shi, Yixuan Wang, To- bias Pfaff, Cheston Tan, Henrik I. Christensen, Hao Su, Jiajun Wu, and Yunzhu Li. A review of learning- based dynamics models for robotic manipulation.Sci- ence Robotics, 10(106):eadt1497, September 2025. doi: 10.1126/scirobotics.adt1497. URL https://www.science. org/doi/10.1126/scirobotics.adt1497

    doi:10.1126/scirobotics.adt1497 2025

  35. [35]

    Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics, April 2025

    Chenhao Li, Andreas Krause, and Marco Hutter. Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics, April 2025. URL http: //arxiv.org/abs/2501.10100. arXiv:2501.10100 [cs]

    arXiv 2025

  36. [36]

    Dexart: Benchmarking generalizable dexterous manipu- lation with articulated objects

    Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. Dexart: Benchmarking generalizable dexterous manipu- lation with articulated objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21190–21200, 2023

    2023

  37. [37]

    Sapien: A simulated part-based interactive environment

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11097–11107, 2020

    2020

  38. [38]

    Robotic grasping and contact: A review

    Antonio Bicchi and Vijay Kumar. Robotic grasping and contact: A review. InProceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065), volume 1, pages 348–353. IEEE, 2000

    2000

  39. [39]

    Learning complex dexterous manipula- tion with deep reinforcement learning and demonstra- tions.arXiv preprint arXiv:1709.10087, 2017

    Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipula- tion with deep reinforcement learning and demonstra- tions.arXiv preprint arXiv:1709.10087, 2017

    Pith/arXiv arXiv 2017

  40. [40]

    Mujoco: A physics engine for model-based control

    Todorov, Emanuel and Erez, Tom and Tassa, Yuval. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012

    2012

  41. [41]

    On the utility of koopman operator theory in learning dexterous manipulation skills

    Yunhai Han, Mandy Xie, Ye Zhao, and Harish Ravichan- dar. On the utility of koopman operator theory in learning dexterous manipulation skills. InConference on Robot Learning, pages 106–126. PMLR, 2023

    2023

  42. [42]

    KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipu- lation, September 2024

    Hongyi Chen, Abulikemu Abuduweili, Aviral Agrawal, Yunhai Han, Harish Ravichandar, Changliu Liu, and Jef- frey Ichnowski. KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipu- lation, September 2024. URL http://arxiv.org/abs/2407. 00548. arXiv:2407.00548 [cs]

    Pith/arXiv arXiv 2024

  43. [43]

    Deep reinforce- ment learning that matters

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforce- ment learning that matters. InProceedings of the AAAI conference on artificial intelligence, 2018

    2018

  44. [44]

    Learning prehensile dexterity by imitating and emulating state-only observations.IEEE Robotics and Automation Letters, 2024

    Yunhai Han, Zhenyang Chen, Kyle A Williams, and Harish Ravichandar. Learning prehensile dexterity by imitating and emulating state-only observations.IEEE Robotics and Automation Letters, 2024

    2024

  45. [45]

    Visu- alizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008

    Laurens van der Maaten and Geoffrey Hinton. Visu- alizing data using t-sne.Journal of machine learning research, 9(Nov):2579–2605, 2008

    2008

  46. [46]

    A closed-form geometric retargeting solver for upper body humanoid robot tele- operation.arXiv preprint arXiv:2602.01632, 2026

    Chuizheng Kong, Yunho Cho, Wonsuhk Jung, Idris Wibowo, Parth Shinde, Sundhar Vinodh-Sangeetha, Long Kiu Chung, Zhenyang Chen, Andrew Mattei, Ad- vaith Nidumukkala, Alexander Elias, Danfei Xu, Taylor Higgins, and Shreyas Kousik. A closed-form geometric retargeting solver for upper body humanoid robot tele- operation.arXiv preprint arXiv:2602.01632, 2026

    arXiv 2026

  47. [47]

    Learning koop- man representations for hybrid systems.arXiv preprint arXiv:2006.12427, 2020

    Craig Bakker, Arnab Bhattacharya, Samrat Chatterjee, Casey J Perkins, and Matthew R Oster. Learning koop- man representations for hybrid systems.arXiv preprint arXiv:2006.12427, 2020

    Pith/arXiv arXiv 2006

  48. [48]

    Desko: Stability-assured robust control with a deep stochastic koopman operator

    Minghao Han, Jacob Euler-Rolle, and Robert K Katzschmann. Desko: Stability-assured robust control with a deep stochastic koopman operator. InInterna- tional conference on learning representations (ICLR), 2022

    2022

  49. [49]

    Mimictouch: Leveraging multi-modal human tactile demonstrations for contact- rich manipulation

    Kelin Yu, Yunhai Han, Qixian Wang, Vaibhav Saxena, Danfei Xu, and Ye Zhao. Mimictouch: Leveraging multi-modal human tactile demonstrations for contact- rich manipulation. In8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id= 7yMZAUkXa4

    2024

  50. [50]

    Neural ordinary differential equations.Advances in neural information processing systems, 31, 2018

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations.Advances in neural information processing systems, 31, 2018

    2018

  51. [51]

    Sam 3: Segment anything with concepts

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoub- hik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025

    Pith/arXiv arXiv 2025

  52. [52]

    Lu Shi and Konstantinos Karydis. Acd-edmd: Analyt- ical construction for dictionaries of lifting functions in koopman operator-based nonlinear robotic systems.IEEE Robotics and Automation Letters, 7(2):906–913, 2021

    2021

  53. [53]

    Koopman operator, geometry, and learning

    Igor Mezic. Koopman operator, geometry, and learning. arXiv preprint arXiv:2010.05377, 2020. APPENDICES VII. DATASETMODIFICATION In typical imitation learning problems, the first actiona (i) 1 varies across demonstrations and is therefore unavailable at in- ference time. To address this issue, UBMs utilize an auxiliary initial actiona (i) 0 = q (i) 1 , wh...

    Pith/arXiv arXiv 2010

  54. [2018]

    doi: 10.1016/j.automatica.2018

    ISSN 0005-1098. doi: 10.1016/j.automatica.2018. 03.046. URL https://www.sciencedirect.com/science/ article/pii/S000510981830133X

    doi:10.1016/j.automatica.2018 2018