Pith. sign in

REVIEW 5 major objections 5 minor 68 references

Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Chain-of-Modality: reading video, force, and hand pose one at a time yields one-shot robot manipulation programs.

desk verdict A sensible step-by-step prompting trick beats single-batch multimodal prompting for extracting manipulation plans from human video, with a plausible but thinly evidenced empirical case. read the letter →

arxiv 2504.13351 v1 pith:N33R4Q4A submitted 2025-04-17 cs.RO cs.AIcs.HCcs.LGcs.MM

Chen Wang , Fei Xia , Wenhao Yu , Tingnan Zhang , Ruohan Zhang , C. Karen Liu , Li Fei-Fei , Jie Tan
show 1 more author
Jacky Liang
This is my paper · ORCID
classification cs.ROcs.AIcs.HCcs.LGcs.MM
keywords chain-of-modalitymultimodalhumanvideovision-languagemodelsrobotmanipulationone-shotimitationEMGforcesignaltaskplanextractionhandposeestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chain-of-Modality is a prompting strategy that asks a vision-language model to read a multimodal human demonstration one modality at a time—force or audio first, then hand pose, then video frames—and to refine a timed, parameterized action plan at each step. The paper's central assertion is that this staged reading lets a single human video teach a robot a manipulation program, including control parameters such as force level and twist direction, which ordinary video cannot convey. Across four tasks the staged prompting reached roughly 60% accuracy in extracting the exact task plan and control parameters, while vision-only prompting scored zero and a single merged multimodal query averaged about 17%. On a real robot, the programs generated from the analysis succeeded in about 73% of trials, including on unseen objects and across two robot platforms. A sympathetic reader should take the contribution as an empirical demonstration that prompting structure, not just model scale, determines whether current vision-language models can use non-visual signals.

What carries the argument

The load-bearing object is the Chain-of-Modality prompt chain: a fixed sequence of vision-language-model queries in which the answer produced from the force/audio stage is concatenated into the hand-pose query, and that combined answer is concatenated into the image query, followed by a code-generation query. The force modality is a single normalized scalar per frame, obtained by taking the maximum over eight EMG channels or the loudness of an audio recording; the hand modality is the 2D pixel locations of the fingertips from a hand-reconstruction model. What this machinery does is force the model to commit to an intermediate reading of each modality before seeing the next, so the final plan is built by refinement rather than by one-shot cross-modal correlation. It also keeps the answer grounded in the literal timing of the force trace, which is where the paper says task segmentation and control parameters come from.

What would settle it

Run the force-analysis stage on new recordings where ground-truth force is measured independently with a calibrated sensor, and compare the model's reported force-onset times, force-offset times, and discrete force levels to the sensor trace. If the force stage cannot locate the same intervals and levels that the sensor records within a few frames, then the claim that the max-pooled EMG or loudness signal is the decisive modality for task segmentation and control parameters is not supported.

Watch

Extended reading notes

Core claim

The discovery the paper is trying to establish is that a vision-language model can perform one-shot imitation of manipulation from a single multimodal human video if the query is decomposed into a chain of per-modality analyses. In the first stage the model reads only the scalar force trace—max-pooled EMG channels or audio loudness—and reports when and how much force is applied. In the second stage it adds the 2D pixel positions of the fingertips, which lets it infer grasps and rotations; in the third stage it adds the RGB frames, which lets it name objects and assemble the full timed action sequence. The same model then compiles that sequence into Python calls against a fixed robot API. The paper reports exact task-plan accuracy of 60% for this chain, compared with zero for image-only input and 17% for a single merged multimodal query, and an average 73% real-robot success rate across opening bottles, inserting plugs, wiping a board, and playing a drum.

Load-bearing premise

The load-bearing premise is that a single scalar force trace (max over eight EMG channels or audio loudness) together with 2D fingertip pixel positions carries enough reliable information for the vision-language model to segment the task and set control parameters; if those derived signals are noisy, ambiguous, or poorly synchronized with the video, the staged analysis inherits the error and the reported accuracy gains would not transfer to new recordings.

Editorial extensions

If this is right

  • The paper's results imply that non-visual channels—muscle activity or impact sound—can supply the control-parameter information that RGB video alone hides, so multimodal capture should be part of any human-video imitation setup for contact-rich tasks.
  • Because each modality is analyzed before the next is added, the gains do not depend on the vision-language model spontaneously aligning long interleaved sequences; the same staged prompt may continue to help as long-context models improve.
  • Generated programs target a platform-agnostic Python API, so a single demonstration video can be re-targeted to different robot arms without retraining; the paper demonstrates this on two bi-manual platforms.
  • The success of the open-vocabulary object-finding API suggests that the bottleneck for video-to-robot transfer is less about low-level trajectory cloning than about extracting the right high-level actions and parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same chain idea could apply to other multimodal reasoning tasks where one modality is weak but informative, such as adding thermal or depth channels to video for inspection workflows, so the prompting pattern is transferable.
  • The paper reduces each EMG burst to a single max value; an editor's test would be whether feeding the per-channel waveform or its spectrogram, instead of the max-pooled scalar, sharpens the model's distinction between force levels beyond the coarse high/medium/low used here.
  • Because robot execution is open-loop, a natural extension the paper leaves implicit is a closed-loop variant that re-runs the analysis when a skill fails, which would test whether the extracted plan or the execution layer is the actual failure point.
  • The zero vision-only accuracy suggests a supervised vision-only force regressor might close part of the gap; the paper does not compare against such a learned baseline, so that comparison remains open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Chain-of-Modality (CoM), a prompting strategy that feeds a multimodal human demonstration video (RGB frames, a scalar force trace derived from EMG or audio, and 2D hand-pose keypoints) to a VLM one modality at a time, progressively refining a task plan and control parameters. A second VLM pass converts the resulting analysis into Python code that calls a predefined robot skill library. The experiments compare CoM with modality-ablation and integration-order baselines on four tasks using Gemini 1.5 Pro and GPT-4o, and evaluate the generated programs on real robots for opening bottles, inserting a plug, wiping a board, and playing a drum, reporting an average robot success rate of 73% against a 92% manually coded oracle.

Significance. If the reported results are reproducible, CoM would be a useful, simple recipe: current VLMs handle interleaved multimodal streams poorly, and sequential per-modality refinement materially improves structured extraction from videos, while the EMG/audio-derived force channel is a sensible way to capture control parameters that images alone cannot provide. The paper has genuine strengths: it tests two modern VLMs, evaluates on two robot embodiments, includes an oracle upper bound, uses ground-truth task plans rather than circular self-evaluation, and makes falsifiable quantitative claims. The main limitation of the current contribution is evidentiary: the central comparisons rest on 10 self-collected videos per task with 3 stochastic queries, no error bars or significance tests, an underspecified exact-match metric, and no robot-level baseline comparison. The prompting idea is plausible and not internally inconsistent, but the evidence base is too thin for the strength of the headline claims.

major comments (5)
  1. [Introduction, §IV.A–IV.B] The headline numbers in the Introduction are not traceable to the reported tables. The Introduction states that CoM achieves 60% accuracy and that naive all-modality batch querying averages 17%, and §IV.B states that CoM outperforms Sep-Sep by more than 19% with Gemini and 17% with GPT-4o. However, Table I contains only Image-only, w.o. img, w.o. force, w.o. hand, and All; the CoM, Merg, Merg-Sep, Sep-Merg, and Sep-Sep rows are not tabulated, and the quantitative source is only the referenced Fig. 5. In addition, the Table I 'All' row averages about 59% for Gemini and 29% for GPT-4o, so it is unclear how the 17% figure relates to the 'All' baseline. The central empirical claim needs a complete table with per-method, per-task, per-model accuracy and similarity values so that the 60%, 17%, and 19%/17% improvements can be verified.
  2. [§IV.A] The evaluation metric is underspecified. The text says the paper reports the 'average success rate in generating the correct human task plans observed in the video, along with the similarity score, which is calculated by finding the longest common string between the output and the ground truth.' It is not stated whether 'accuracy' means exact string equality, whether it is computed per action, per video, or per query, how the three queries per video are aggregated (any correct, majority vote, or mean), or how the ground-truth task plans were obtained and annotated. The phrase 'longest common string' is also ambiguous between longest common substring and longest common subsequence. These choices materially affect Table I and the claimed 60%; the authors should provide the exact evaluation protocol, ideally with the evaluation code.
  3. [§III.A, §III.D, Conclusion] The scalar force/audio summaries on which the whole method depends are not validated. Section III.D defines the force signal as the per-timestep maximum across eight EMG channels sampled at 200 Hz and downsampled to 60 Hz, and audio as loudness; Sections III.A–III.B then use the timing and level of force as key cues for segmentation and for control parameters such as twist angle and force intensity. The paper provides no evidence that the max-pooled EMG envelope reliably marks grasp/release events and relative effort, no per-subject or per-session analysis, and no failure analysis for HaMeR 2D fingertip tracks under occlusion or viewpoint change. If muscle co-activation, background audio, or projection-dependent finger motion corrupts these traces, CoM's chain will inherit the error and the 10-video benchmark results may not transfer to new recordings. The conclusion lists only the audio-volume limitation, not the force-envelope preprocessing; please add a calibration or control experiment that aligns envelope peaks with independently annotated contact/release events and reports the force-only stage's accuracy.
  4. [§IV.B, Table II] The on-robot evaluation does not test CoM's central advantage. Table II reports only 'Ours' and an oracle program; no baseline-generated code (for example, from Image-only or Merged analysis) is executed on the robot. The 73% average success rate therefore establishes absolute feasibility, not that CoM's prompting advantage survives to execution, and the claim of 'strong generalization' is not a comparative claim on the current data. The authors should either add robot-level baseline comparisons in the same 20-trial generalization settings, or explicitly restrict the generalization claim to absolute performance.
  5. [§IV.A, Table I] The statistical basis is too thin for the strength of the claims. Each task uses 10 self-collected videos and 3 queries per method, and Table I contains no confidence intervals, no per-video breakdown, and no significance tests. With n=10 and three stochastic samples, differences such as GPT-4o's 0.00 versus 0.40 on Pressing Cube are not established as beyond sampling noise. Report per-video results with confidence intervals or significance tests, or temper the quantitative claims accordingly.
minor comments (5)
  1. [§IV.B, Figs. 4–5] Figure references are inconsistent: the text in §IV.B refers to 'Fig. 5, first column' for qualitative Opening Bottle results, while Fig. 4 is the qualitative figure and Fig. 5 is the quantitative comparison; please renumber or fix the references.
  2. [§III.D, §IV.A] The hand-pose input is described inconsistently: §III.D says HaMeR localizes 'the pixel locations of the fingertips', while §IV.A says the hand pose data 'consists of 2D pixel locations of the thumb and middle fingertips'; specify which fingers are actually provided to the VLM.
  3. [§IV.A–IV.B] The sentence 'This video demonstrates only the key features of primitive skills, which does not include the testing objects or task plans' appears twice, once in the Prompts paragraph and once in the Results paragraph; remove the duplication.
  4. [Table I] The caption says 'Accuracy / Similarity Score'; please add a legend stating which column is which and include the number of videos and queries per method in the caption.
  5. [Throughout] Minor writing issues: 'w.o. img' and 'w.o. image' are used interchangeably, 'as Fig. 3(c) illustrated' should be 'as illustrated in Fig. 3(c)', and 'Gemini 1.5 pro' should be 'Gemini 1.5 Pro' for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CoM's claims are empirical and grounded in external ground-truth plans and held-out robot trials.

full rationale

The paper's central claims are empirical evaluations of a prompting strategy, not derivations from fitted parameters or self-cited theorems. The prompt's single example video is explicitly task-irrelevant ('the human is pressing and rotating an apple and a can on the table. This video demonstrates only the key features of primitive skills, which does not include the testing objects or task plans'), so the output format example does not encode the evaluation answers. Accuracy is measured against external ground-truth task plans, and all baselines share the same example video and input modalities. The force signal (max over eight EMG channels), audio loudness, and HaMeR hand poses are fixed preprocessing choices, not parameters tuned to the benchmark outcomes. Robot success is measured on unseen objects, placements, and beats in held-out trials, with an Oracle upper bound reported separately. The few self-citations (e.g., Code as Policies for generating API calls) are used only to motivate the code-generation component and do not supply the load-bearing evidence for CoM's accuracy claims. No equation or fitted parameter is renamed as a prediction, and no uniqueness or equivariance result is imported from the authors' prior work. The evaluation is self-contained against external benchmarks, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method introduces no free parameters or invented entities; its action library, prompt templates, and signal preprocessing are hand-designed but not fitted to the evaluation. The claims rest on domain assumptions about sensor signal informativeness, VLM reasoning reliability, and the sufficiency of the open-loop API set.

assumptions (4)
  • domain assumption The EMG/audio-derived single-scalar force signal reliably indicates when and how hard the human applies force during manipulation.
    Invoked in Section III.A and III.D; the force signal is max-pooled over eight EMG channels or computed as audio loudness, and CoM's first stage uses its peaks to segment subtasks. If this signal is noisy, the segmentation rationale fails.
  • domain assumption Vision-language models can reason correctly over numerical signal arrays and 2D hand pose coordinates when prompted sequentially.
    The entire method assumes Gemini 1.5 Pro and GPT-4o can map the provided scalar and coordinate inputs to the intended actions; this is an empirical property of the models, not a proven guarantee.
  • domain assumption The hand-designed action library and robot API set are expressive enough to represent the demonstrated tasks.
    The VLM is constrained to output actions from a pre-specified library (Grasp, Release, Twist, Find, Move_to, Insert, Wipe, Hit). The paper does not analyze failure cases where the true task falls outside this library.
  • domain assumption HaMeR hand pose estimates provide accurate enough fingertip positions for inferring grasp and twist parameters.
    Section III.A treats fingertip pixel locations as an input modality; the Opening Bottle task relies on these to infer 180-degree twist directions. Accuracy of HaMeR on the recorded videos is not quantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models." pith.science (2026). https://pith.science/paper/N33R4Q4A

@misc{pith2026250413351,
  author       = {Pith},
  title        = {Pith review of: Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N33R4Q4A}},
  note         = {Machine review of arXiv:2504.13351}
}
read the original abstract

Learning to perform manipulation tasks from human videos is a promising approach for teaching robots. However, many manipulation tasks require changing control parameters during task execution, such as force, which visual data alone cannot capture. In this work, we leverage sensing devices such as armbands that measure human muscle activities and microphones that record sound, to capture the details in the human manipulation process, and enable robots to extract task plans and control parameters to perform the same task. To achieve this, we introduce Chain-of-Modality (CoM), a prompting strategy that enables Vision Language Models to reason about multimodal human demonstration data -- videos coupled with muscle or audio signals. By progressively integrating information from each modality, CoM refines a task plan and generates detailed control parameters, enabling robots to perform manipulation tasks based on a single multimodal human video prompt. Our experiments show that CoM delivers a threefold improvement in accuracy for extracting task plans and control parameters compared to baselines, with strong generalization to new task setups and objects in real-world robot experiments. Videos and code are available at https://chain-of-modality.github.io

Figures

Figures reproduced from arXiv: 2504.13351 by the authors.

Figure 1
Figure 1. We introduce Chain-of-Modality (CoM), a prompting [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of Chain-of-Modality (using force as an example). [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of Experiment Tasks. (a) Multimodal Human Video Input: Our framework processes a single-shot human video with force or audio data, using Chain-of-Modality to extract the task plan and control parameters, then generates a robot control program. (b) Robot Code Execution: The robot executes the program to replicate the task observed in the video. (c) Evaluation Setups: We evaluate the performance of generated … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative results for Chain-of-Modality. We showcase task plans generated by CoM for four evaluation videos. CoM [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Quantitative results for Chain-of-Modality. We compare CoM with baselines across three tasks using both Gemini and GPT. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 37 canonical work pages

  1. [1]

    Gpt-4 technical report,

    J. Achiam et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Do as i can, not as i say: Ground- ing language in robotic affordances,

    M. Ahn et al. , “Do as i can, not as i say: Ground- ing language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022

    arXiv 2022

  3. [3]

    Human-to-robot imitation in the wild,

    S. Bahl, A. Gupta, and D. Pathak, “Human-to-robot imitation in the wild,” arXiv preprint arXiv:2207.09450 , 2022

    arXiv 2022

  4. [4]

    Affordances from human videos as a versatile representation for robotics,

    S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak, “Affordances from human videos as a versatile representation for robotics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 778–13 790

    2023

  5. [5]

    Towards generalizable zero-shot manipulation via translating human interaction plans,

    H. Bharadhwaj, A. Gupta, V . Kumar, and S. Tul- siani, “Towards generalizable zero-shot manipulation via translating human interaction plans,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 6904–6911

    work page 2024

  6. [6]

    Activitynet: A large-scale video benchmark for human activity understanding,

    F. Caba Heilbron, V . Escorcia, B. Ghanem, and J. Carlos Niebles, “Activitynet: A large-scale video benchmark for human activity understanding,” in Proceedings of the ieee conference on computer vision and pattern recognition, 2015, pp. 961–970

    work page 2015

  7. [7]

    Procedure planning in instructional videos,

    C.-Y . Chang, D.-A. Huang, D. Xu, E. Adeli, L. Fei-Fei, and J. C. Niebles, “Procedure planning in instructional videos,” in European Conference on Computer Vision , Springer, 2020, pp. 334–350

    work page 2020

  8. [8]

    Learning generalizable robotic reward functions from

    A. S. Chen, S. Nair, and C. Finn, “Learning generalizable robotic reward functions from” in-the-wild” human videos,” Robotics: Science and Systems (RSS) , 2021

    work page 2021

Show all 68 references
  1. [9]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,

    B. Chen et al., “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 455–14 465

    2024

  2. [10]

    Mobility vla: Multimodal instruc- tion navigation with long-context vlms and topological graphs,

    H.-T. L. Chiang et al., “Mobility vla: Multimodal instruc- tion navigation with long-context vlms and topological graphs,” arXiv preprint arXiv:2407.07775 , 2024

    2024 arXiv

  3. [11]

    Can foundation models perform zero-shot task specification for robot manipulation?

    Y . Cui, S. Niekum, A. Gupta, V . Kumar, and A. Rajeswaran, “Can foundation models perform zero-shot task specification for robot manipulation?” In Learning for dynamics and control conference , PMLR, 2022, pp. 893–905

    2022

  4. [12]

    Scaling egocentric vision: The epic- kitchens dataset,

    D. Damen et al., “Scaling egocentric vision: The epic- kitchens dataset,” in European Conference on Computer Vision (ECCV), 2018

    2018

  5. [13]

    Model-based inverse reinforcement learning from visual demonstrations,

    N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier, “Model-based inverse reinforcement learning from visual demonstrations,” in Conference on Robot Learning, PMLR, 2021, pp. 1930–1942

    2021

  6. [14]

    Perceptual values from observation,

    A. D. Edwards and C. L. Isbell, “Perceptual values from observation,” arXiv preprint arXiv:1905.07861 , 2019

    1905 arXiv

  7. [15]

    Foundation models in robotics: Applications, challenges, and the future,

    R. Firoozi et al. , “Foundation models in robotics: Applications, challenges, and the future,” arXiv preprint arXiv:2312.07843, 2023

    2023 arXiv

  8. [16]

    The” something something

    R. Goyal et al. , “The” something something” video database for learning and evaluating visual common sense,” in Proceedings of the IEEE international confer- ence on computer vision , 2017, pp. 5842–5850

    2017

  9. [17]

    Ego4d: Around the world in 3,000 hours of egocentric video,

    K. Grauman et al. , “Ego4d: Around the world in 3,000 hours of egocentric video,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 995–19 012

    2022

  10. [18]

    Gu et al., Rt-trajectory: Robotic task generalization via hindsight trajectory sketches , 2023

    J. Gu et al., Rt-trajectory: Robotic task generalization via hindsight trajectory sketches , 2023. arXiv: 2311. 01977 [cs.RO]

    2023

  11. [19]

    Toward general-purpose robots via foundation models: A survey and meta-analysis,

    Y . Hu et al. , “Toward general-purpose robots via foundation models: A survey and meta-analysis,” arXiv preprint arXiv:2312.08782, 2023

    2023 arXiv

  12. [20]

    Neural task graphs: Generalizing to unseen tasks from a single video demonstration,

    D.-A. Huang et al., “Neural task graphs: Generalizing to unseen tasks from a single video demonstration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 8565–8574

    2019

  13. [21]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in Interna- tional conference on machine learning , PMLR, 2022, pp. 9118–9147

    2022

  14. [22]

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei, “Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,” arXiv preprint arXiv:2409.01652, 2024

    2024 arXiv

  15. [23]

    Inner monologue: Embodied reasoning through planning with language models,

    W. Huang et al., “Inner monologue: Embodied reasoning through planning with language models,” arXiv preprint arXiv:2207.05608, 2022

    2022 arXiv

  16. [24]

    Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers,

    V . Jainet al., “Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers,” arXiv preprint arXiv:2403.12943, 2024

    2024 arXiv

  17. [25]

    Prompting visual-language models for efficient video understanding,

    C. Ju, T. Han, K. Zheng, Y . Zhang, and W. Xie, “Prompting visual-language models for efficient video understanding,” in European Conference on Computer Vision, Springer, 2022, pp. 105–124

    2022

  18. [26]

    Human action recognition and prediction: A survey,

    Y . Kong and Y . Fu, “Human action recognition and prediction: A survey,” International Journal of Computer Vision, vol. 130, no. 5, pp. 1366–1401, 2022

    2022

  19. [27]

    Anticipating human activities using object affordances for reactive robotic response,

    H. S. Koppula and A. Saxena, “Anticipating human activities using object affordances for reactive robotic response,” IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 1, pp. 14–29, 2015

    2015

  20. [28]

    Graph inverse reinforcement learning from diverse videos,

    S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang, “Graph inverse reinforcement learning from diverse videos,” in Conference on Robot Learning, PMLR, 2023, pp. 55–66

    2023

  21. [29]

    Hake: Human activity knowledge engine,

    Y .-L. Li et al., “Hake: Human activity knowledge engine,” arXiv preprint arXiv:1904.06539 , 2019

    1904 arXiv

  22. [30]

    Code as policies: Language model programs for embodied control,

    J. Liang et al. , “Code as policies: Language model programs for embodied control,” in 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA) , IEEE, 2023, pp. 9493–9500

    2023

  23. [31]

    Learning to learn faster from human feedback with language model predictive control,

    J. Liang et al., “Learning to learn faster from human feedback with language model predictive control,” arXiv preprint arXiv:2402.11450, 2024

    2024 arXiv

  24. [32]

    Text2motion: From natural language instructions to feasible plans,

    K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, “Text2motion: From natural language instructions to feasible plans,” Autonomous Robots , vol. 47, no. 8, pp. 1345–1365, 2023

    2023

  25. [33]

    Llm+ p: Empowering large language mod- els with optimal planning proficiency,

    B. Liu et al., “Llm+ p: Empowering large language mod- els with optimal planning proficiency,” arXiv preprint arXiv:2304.11477, 2023

    2023 arXiv

  26. [34]

    Imitation from observation: Learning to imitate behaviors from raw video via context translation,

    Y . Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observation: Learning to imitate behaviors from raw video via context translation,” in 2018 IEEE international conference on robotics and automation (ICRA) , IEEE, 2018, pp. 1118–1125

    2018

  27. [35]

    Vip: Towards universal visual reward and representation via value-implicit pre-training,

    Y . J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V . Kumar, and A. Zhang, “Vip: Towards universal visual reward and representation via value-implicit pre-training,” arXiv preprint arXiv:2210.00030 , 2022

    2022 arXiv

  28. [36]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms,

    S. Nasiriany et al., “Pivot: Iterative visual prompting elicits actionable knowledge for vlms,” arXiv preprint arXiv:2402.07872, 2024

    2024 arXiv

  29. [37]

    R+ x: Retrieval and execution from everyday human videos,

    G. Papagiannis, N. Di Palo, P. Vitiello, and E. Johns, “R+ x: Retrieval and execution from everyday human videos,” arXiv preprint arXiv:2407.12957 , 2024

    2024 arXiv

  30. [38]

    Reconstructing hands in 3D with transformers,

    G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik, “Reconstructing hands in 3D with transformers,” in CVPR, 2024

    2024

  31. [39]

    Planning with large language models via corrective re-prompting,

    S. S. Raman, V . Cohen, E. Rosen, I. Idrees, D. Paulius, and S. Tellex, “Planning with large language models via corrective re-prompting,” in NeurIPS 2022 Foundation Models for Decision Making Workshop , 2022

    2022

  32. [40]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    M. Reid et al. , “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530 , 2024

    2024 arXiv

  33. [41]

    First-person activity forecasting with online inverse reinforcement learning,

    N. Rhinehart and K. M. Kitani, “First-person activity forecasting with online inverse reinforcement learning,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 3696–3705

    2017

  34. [42]

    Reinforcement learning with videos: Combining offline observations with interaction,

    K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn, “Reinforcement learning with videos: Combining offline observations with interaction,” arXiv preprint arXiv:2011.06507, 2020

    2011 arXiv

  35. [43]

    Learning predictive models from observation and interaction,

    K. Schmeckpeper et al., “Learning predictive models from observation and interaction,” in European Confer- ence on Computer Vision , Springer, 2020, pp. 708–725

    2020

  36. [44]

    Time-contrastive networks: Self- supervised learning from video,

    P. Sermanet et al. , “Time-contrastive networks: Self- supervised learning from video,” in 2018 IEEE interna- tional conference on robotics and automation (ICRA) , IEEE, 2018, pp. 1134–1141

    2018

  37. [45]

    Unsupervised perceptual rewards for imitation learning,

    P. Sermanet, K. Xu, and S. Levine, “Unsupervised perceptual rewards for imitation learning,” arXiv preprint arXiv:1612.06699, 2016

    2016 arXiv

  38. [46]

    Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,

    D. Shah, B. Osi ´nski, S. Levine, et al., “Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,” in Conference on robot learning , PMLR, 2023, pp. 492–504

    2023

  39. [47]

    Concept2robot: Learning manipulation concepts from instructions and human demonstrations,

    L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg, “Concept2robot: Learning manipulation concepts from instructions and human demonstrations,” The International Journal of Robotics Research , vol. 40, no. 12-14, pp. 1419–1434, 2021

    2021

  40. [48]

    Third-person visual imitation learning via decoupled hierarchical controller,

    P. Sharma, D. Pathak, and A. Gupta, “Third-person visual imitation learning via decoupled hierarchical controller,” Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  41. [49]

    Videodex: Learning dexterity from internet videos,

    K. Shaw, S. Bahl, and D. Pathak, “Videodex: Learning dexterity from internet videos,” CoRL, 2022

    2022

  42. [50]

    Cliport: What and where pathways for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in Con- ference on robot learning , PMLR, 2022, pp. 894–906

    2022

  43. [51]

    Generalized planning in pddl domains with pretrained large language models,

    T. Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L. Kaelbling, and M. Katz, “Generalized planning in pddl domains with pretrained large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 2024, pp. 20 256–20 264

    2024

  44. [52]

    Progprompt: Generating situated robot task plans using large language models,

    I. Singh et al., “Progprompt: Generating situated robot task plans using large language models,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2023, pp. 11 523–11 530

    2023

  45. [53]

    Avid: Learning multi-stage tasks via pixel- level translation of human videos,

    L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine, “Avid: Learning multi-stage tasks via pixel- level translation of human videos,” arXiv preprint arXiv:1912.04443, 2019

    1912 arXiv

  46. [54]

    Rt-sketch: Goal-conditioned imitation learning from hand-drawn sketches,

    P. Sundaresan et al. , “Rt-sketch: Goal-conditioned imitation learning from hand-drawn sketches,” 2024

    2024

  47. [55]

    Mimicplay: Long-horizon imitation learning by watching human play,

    C. Wang et al. , “Mimicplay: Long-horizon imitation learning by watching human play,” arXiv preprint arXiv:2302.12422, 2023

    2023 arXiv

  48. [56]

    Temporal segment networks for action recognition in videos,

    L. Wang et al., “Temporal segment networks for action recognition in videos,” IEEE transactions on pattern analysis and machine intelligence , vol. 41, no. 11, pp. 2740–2755, 2018

    2018

  49. [57]

    Describe, explain, plan and select: Interactive planning with large language models enables open-world multi- task agents,

    Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, and Y . Liang, “Describe, explain, plan and select: Interactive planning with large language models enables open-world multi- task agents,” arXiv preprint arXiv:2302.01560 , 2023

    2023 arXiv

  50. [58]

    You only demonstrate once: Category-level manipulation from single visual demonstration,

    B. Wen, W. Lian, K. Bekris, and S. Schaal, “You only demonstrate once: Category-level manipulation from single visual demonstration,” arXiv preprint arXiv:2201.12716, 2022

    2022 arXiv

  51. [59]

    Wen et al., Any-point trajectory modeling for policy learning, 2023

    C. Wen et al., Any-point trajectory modeling for policy learning, 2023. arXiv: 2401.00025 [cs.RO]

    2023 arXiv

  52. [60]

    Learning by watching: Physical imitation of manipulation skills from human videos,

    H. Xiong, Q. Li, Y .-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg, “Learning by watching: Physical imitation of manipulation skills from human videos,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , IEEE, 2021, pp. 7827–7834

    2021

  53. [61]

    R-c3d: Region convolutional 3d network for temporal activity detection,

    H. Xu, A. Das, and K. Saenko, “R-c3d: Region convolutional 3d network for temporal activity detection,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 5783–5792

    2017

  54. [62]

    Xskill: Cross embodiment skill discovery,

    M. Xu, Z. Xu, C. Chi, M. Veloso, and S. Song, “Xskill: Cross embodiment skill discovery,” in Conference on Robot Learning, PMLR, 2023, pp. 3536–3555

    2023

  55. [63]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision),

    Z. Yang et al. , “The dawn of lmms: Preliminary explorations with gpt-4v (ision),” arXiv preprint arXiv:2309.17421, vol. 9, no. 1, p. 1, 2023

    2023 arXiv

  56. [64]

    Language to rewards for robotic skill synthesis,

    W. Yu et al. , “Language to rewards for robotic skill synthesis,” arXiv preprint arXiv:2306.08647 , 2023

    2023 arXiv

  57. [65]

    Xirl: Cross-embodiment inverse rein- forcement learning,

    K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi, “Xirl: Cross-embodiment inverse rein- forcement learning,” in Conference on Robot Learning , PMLR, 2022, pp. 537–546

    2022

  58. [66]

    Socratic models: Composing zero-shot multimodal reasoning with language,

    A. Zeng et al., “Socratic models: Composing zero-shot multimodal reasoning with language,” arXiv preprint arXiv:2204.00598, 2022

    2022 arXiv

  59. [67]

    Actionformer: Localizing moments of actions with transformers,

    C.-L. Zhang, J. Wu, and Y . Li, “Actionformer: Localizing moments of actions with transformers,” in European Conference on Computer Vision, Springer, 2022, pp. 492– 510

    2022

  60. [68]

    Vision-based manipulation from single human video with open-world object graphs,

    Y . Zhu, A. Lim, P. Stone, and Y . Zhu, “Vision-based manipulation from single human video with open-world object graphs,” arXiv preprint arXiv:2405.20321 , 2024

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.