Pith. sign in

REVIEW 5 major objections 6 minor 60 references

Learning Human Skill Generators at Key-Step Levels

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces KS-Gen, a task and framework for generating key-step video clips that complete a human skill from one initial image and a skill description, and reports that its Key-step Image Generator improves image quality and…

desk verdict A solid systems paper that defines a genuinely new task and a useful benchmark, but its headline claim is conditional on an evaluation that doesn't yet prove generated clips follow the right steps. read the letter →

arxiv 2502.08234 v1 pith:6KH7D2ET submitted 2025-02-12 cs.CV

Yilu Wu , Chenhui Zhu , Shuai Wang , Hanlin Wang , Jing Wang , Zhaoxiang Zhang , Limin Wang This is my paper · ORCID
classification cs.CV
keywords key-stepskillgenerationproceduralvideomultimodalLLMplanningretrieval-augmentedimageinstructionaldatasetsdiffusionmodelsdatacurationpipeline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a task called Key-step Skill Generation: given one image of a starting state and a short skill description, produce short video clips of each key step needed to complete the skill, instead of one long continuous video. The authors argue this matches how instructional videos are actually filmed and sidesteps the redundant segments that make long-video generation hard. They support the task with a curated dataset of roughly 110,000 subclips from existing instructional-video sources, a set of quality and similarity metrics, and a three-part generation framework. The framework plans step descriptions with a retrieval-augmented multimodal language model, draws each step's opening frame with a new Key-step Image Generator, and animates the clips with fine-tuned video diffusion models. The central claim is that this pipeline generates coherent key-step skill videos and that the Key-step Image Generator improves image quality and consistency; if true, it gives both human learners and embodied agents a practical source of step-level skill demonstrations.

What carries the argument

The load-bearing object is the Key-step Image Generator (KIG), a two-part module. A causal Transformer decoder, called the Skill Transformer, takes the IP-Adapter image features of the initial image and CLIP text features of each step description, and autoregressively predicts the image features of each later step's opening frame; those predicted features are fused with the initial-image features and injected through IP-Adapter into a frozen text-to-image diffusion model to synthesize the step image. The video generator then animates each step image with its description. This machinery replaces the missing first frames of later clips, so the pipeline does not need autoregressive video continuation across scene transitions, and it keeps step images consistent with both the initial state and the textual step.

What would settle it

Take a test skill and deliberately swap the order of two generated key-step clips while keeping the same images and prompts; if the Action, CLIP, DINO, Motion, FVD, and FID scores stay at the same level for the wrong order as for the correct order, then the metrics cannot detect whether the skill procedure is correctly generated, and the central claim would fall back to visual plausibility rather than procedural correctness.

Watch

Extended reading notes

Core claim

On the paper's terms, the discovery is that a complex human skill can be generated as a sequence of non-continuous key-step clips by separating procedure planning from pixel synthesis. The step plan comes from a multimodal LLM stabilized with retrieval from a skill database; the visual bridge between steps comes from the Key-step Image Generator, which autoregressively predicts IP-Adapter image features for each later step from the initial image and step texts and feeds them into a frozen text-to-image diffusion model; and each clip is animated by a video generation model conditioned on that step's image and description. Experiments across several video models and ablations report that this decomposition beats direct baseline image generators on CLIP, DINO, FID, and FVD, and that fine-tuned video models outperform zero-shot ones. The paper also reports that retrieval-augmented planning raises exact full-sequence success from 17 percent to 50.5 percent in a close-set setting.

Load-bearing premise

The evaluation assumes that feature-similarity scores between generated clips and real test clips measure whether the skill's steps were generated correctly, even though the planner only gets about half of full step sequences exactly right and the user study covers only ten examples per aspect.

Editorial extensions

If this is right

  • Skills can be rendered as a few short clips rather than one long video, which is closer to real instructional footage and reduces redundant content.
  • The Key-step Image Generator improves image-level consistency and quality over using prior image generators alone, as measured by CLIP, DINO, and FID, and by the user-study correlations.
  • Fine-tuning video diffusion models on curated two-second subclips substantially improves generation over zero-shot use, while the zero-shot model remains competitive.
  • Each data-curation filter, including scene-transition removal, motion filtering, semantic alignment, and text-heavy removal, contributes measurable improvements to the final generator.
  • Retrieval-augmented planning lifts exact step-sequence success from 17.1 percent to 50.5 percent and step accuracy from 37.7 percent to 65.0 percent, showing that a skill database can discipline LLM planners.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the planner still gets only about half of full step sequences exactly right, the high video-similarity scores may partly reflect visually generic actions; a direct test of whether generated clips match their step descriptions would separate planning errors from rendering quality.
  • If KIG's feature prediction transfers across image models, the step-image stage could be upgraded by swapping the frozen diffusion backbone without retraining the Skill Transformer, making the framework modular across image generators.
  • The same decomposition, plan steps, draw a key frame per step, animate, could be applied to procedural domains such as assembly, repair, or cooking, where state changes per step are visually distinct; skills with invisible state changes would need additional conditioning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces KS-Gen, a task in which a model must produce a sequence of key-step video clips from only an initial image and a skill description. The authors build a curated benchmark from COIN, CrossTask, HT-Step, and Kinetics-400, and propose a three-stage framework: an MLLM-based step planner with retrieval augmentation, a Key-step Image Generator (KIG) that autoregressively produces first-frame images for each step, and fine-tuned video diffusion models that turn these images and step descriptions into short clips. The main claims are that this framework generates coherent key-step skill videos and that KIG improves both image quality and consistency, supported by tables of Action, CLIP, DINO, Motion, FVD, and FID scores, ablations over image/video generators and data-curation steps, and a user study.

Significance. If the evaluation were strengthened, the paper would make a useful contribution: the key-step formulation is a sensible reduction of long-horizon skill generation, the data curation pipeline is detailed and appears to help across metrics, and KIG is lightweight (5 GPU hours, 11GB VRAM) while improving several image and video metrics. The authors also release code and data, and they include extensive ablations that map the design space. However, the primary claim of correctly generating human skills is not yet supported. The planner reaches only 65.0% individual-step accuracy and 50.5% full-sequence success even with retrieval in the close-set setting, yet the headline evaluations do not separate videos whose planned step was correct from those where it was not. The current metrics measure visual similarity to reference clips, not whether the planned key step is the correct one, so the paper currently demonstrates visually plausible clips rather than procedurally correct skills.

major comments (5)
  1. [Section 3.3, Tables 1–5, Table 6] The evaluation protocol compares generated clips to the ground-truth reference clip for each key step using feature-similarity metrics (Action, CLIP, DINO, Motion, FVD, FID). These metrics measure visual/feature similarity, not whether the generated step is the correct key step. With the ChatGPT-4o-latest+RAG planner reaching only 65.0% step accuracy and 50.5% full-sequence success (Table 6), roughly one-third of generated clips correspond to a different step than the reference, yet CLIP/DINO can still be high because scene and objects overlap, and the Action score is a cosine similarity between VideoMAE logit vectors rather than a step-classification score. No results are reported separately for clips whose planned step was correct versus incorrect, so Table 1 cannot distinguish 'the skill was generated correctly' from 'a visually fluent video was produced for an incorrect step.' Please add confidence intervals or significance tests and, crucially, report the main metrics conditioned on planning correctness.
  2. [Section 4.1, Table 6] The retrieval database for planning is built from the same COIN and CrossTask sources that contain the test skills. In the close-set setting this means the top-3 retrieved examples can include the ground-truth step sequence for the test skill; even then SR is only 50.5%. Without retrieval, SR drops to 17.1%. Since the open-set results are only qualitative (Figures 7 and 11), the planning component's generalization is not quantitatively demonstrated. Please report quantitative open-set planning accuracy or at least add a clear statement that open-set generation is not evaluated quantitatively.
  3. [Appendix D, Figure 9] The user study contains only 10 examples per aspect. The strong correlation coefficients (e.g., r=0.976 for DINO) are computed over four model-level points and are therefore not evidence that the metrics reliably track human judgments of procedural correctness across the skill distribution; the study also focuses on action accuracy and object consistency within a clip, not on whether the whole multi-step procedure is correct. Please enlarge the study or temper the claim in Section 5.2 that 'these metrics effectively capture human preferences.'
  4. [Section 4.1, Table 8] The final planner is ChatGPT-4o-latest, a proprietary model whose version and behavior are outside the authors' control; the reported planning numbers therefore have a reproducibility risk. Please release the exact prompts, retrieval database, and step pool construction, and consider adding at least one open-weights MLLM configuration to the main results.
  5. [Section 5.1, Tables 1 and 2] The claim that KIG improves 'both quality and consistency' is only indirectly supported. Table 2 shows CLIP/DINO/FID gains for image generation, and Table 1 shows downstream gains, but no dedicated consistency metric (e.g., object-state correctness or cross-clip object identity) is reported. Moreover, some differences in Table 1 are within the range one would expect from noise without error bars: DC+Ours ties DC+Gen on Action (40.8 vs 40.8), and Cog+Ours has a worse Motion score than Cog+SDXL (3.11 vs 3.06). Please report paired differences with significance tests and, if possible, a consistency metric that directly targets the claimed benefit.
minor comments (6)
  1. [Section 3.2] The phrase 'about about 110,000 subclips' contains a duplicated word; please correct it.
  2. [Section 4.1 and Figure 2 caption] The term 'retrieval argument' should be 'retrieval augmentation' wherever it appears, including Figure 2, Table 6, and Section 4.1 caption text.
  3. [Section 5.2] The sentence 'As shown in Figure 3' in the video generation model ablation refers to model ablation results; it should refer to Table 3, while Figure 3 illustrates the KIG architecture.
  4. [Abstract] The abstract contains 'can synthesis simple and atomic human operations'; 'synthesis' should be 'synthesize'.
  5. [Section 4.2] The phrase 'To address the challenge of missing {I1, I2, ..., In−1} and and the inability' contains a duplicated 'and'; please correct it.
  6. [Section 3.2, Table 7] The text states the training set records 'over 28,500 operational videos', while Table 7 lists 29,116 total videos including the test set; please reconcile these numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the pipeline is evaluated against external reference videos and pretrained feature models, with no prediction reducing to a fitted input.

full rationale

The paper makes no mathematical derivation; its claims are empirical benchmark results. The central pipeline (MLLM planning, KIG, video generation) is evaluated against external reference clips with pretrained feature models (VideoMAE, CLIP, DINO, RAFT, FID/FVD), and none of the reported metrics is constructed from the paper's own fitted values. The only author-self connection is VideoMAE used for the Action score; that model is pretrained on Something-Something v2 and is not fitted to KS-Gen test data, so it constitutes independent evidence rather than circularity. KIG is trained with MSE losses against ground-truth IP-Adapter features and at inference predicts features from the initial image, but the final images are produced by a frozen SDXL and judged by external similarity metrics, so the prediction is not the training target renamed. The retrieval database and close-set step pool are drawn from the same COIN/CrossTask sources as the test set, which can inflate planner scores, but this is a disclosed experimental design choice, not a derivation that assumes its conclusion. The low planner success rate (50.5% SR and 65.0% Acc in Table 6) is a correctness and validity concern for the headline claim, not a circularity concern, because the main tables do not claim the planner is perfect and the metrics still measure visual similarity to real clips. No load-bearing step reduces by construction to its own inputs, and no central conclusion rests on a self-citation chain. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical entities or conserved quantities are introduced; KIG is a learned module, not a postulated entity. The main free parameters are the KIG fusion and consistency weights, the curation thresholds (numerical values unreported), and the retrieval top-k. The assumptions are domain-level premises about annotations, feature sufficiency, metric validity, and teacher-forcing transfer, all of which are plausible but not independently verified by the paper.

free parameters (5)
  • Fusion weight w = 0.5
    Combines predicted IP-Adapter feature with initial image feature in KIG inference (Appendix F). Chosen by hand; directly affects all generated key-step images.
  • Consistency weight = 0.5
    Balances the two MSE losses in Skill Transformer training (Appendix F). Chosen by hand.
  • Data curation thresholds
    Scene-transition L1 thresholds, optical-flow KL threshold, EVAL-CLIP alignment threshold, and text-coverage threshold are described qualitatively (Section 3.2, Appendix B) but exact values are omitted; they control which subclips enter the training set.
  • Retrieval top-k examples = 3
    Number of similar skill sequences retrieved to prompt the MLLM planner (Section 4.1). Chosen by hand.
  • Number of steps T = ground-truth per skill
    In close-set evaluation, the number of key steps is pre-defined, which simplifies the planning problem (Section 4.1).
assumptions (4)
  • domain assumption Manual step annotations in COIN, CrossTask, and HT-Step define correct key steps for each skill.
    The dataset construction uses these annotations as ground truth for key-step segmentation and test evaluation (Section 3.2).
  • domain assumption IP-Adapter image features are a sufficient representation for autoregressive prediction of future step images; predicted features fused with the initial feature retain semantic consistency.
    KIG's design assumes that causal Transformer prediction in feature space followed by SDXL decoding yields coherent images (Section 4.2, Figure 3).
  • domain assumption Feature-similarity metrics (CLIP, DINO, Action, FVD, etc.) against real clips are valid proxies for skill generation quality.
    All main results rely on these metrics (Section 3.3); the user study covers only 10 examples per aspect and the correlation is shown without uncertainty (Appendix D).
  • domain assumption Teacher forcing with ground-truth features transfers to autoregressive inference.
    KIG is trained with ground-truth features but applied to self-predicted features at test time (Section 4.2), a standard exposure-bias assumption not validated for this feature space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Human Skill Generators at Key-Step Levels." pith.science (2026). https://pith.science/paper/6KH7D2ET

@misc{pith2026250208234,
  author       = {Pith},
  title        = {Pith review of: Learning Human Skill Generators at Key-Step Levels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KH7D2ET}},
  note         = {Machine review of arXiv:2502.08234}
}
read the original abstract

We are committed to learning human skill generators at key-step levels. The generation of skills is a challenging endeavor, but its successful implementation could greatly facilitate human skill learning and provide more experience for embodied intelligence. Although current video generation models can synthesis simple and atomic human operations, they struggle with human skills due to their complex procedure process. Human skills involve multi-step, long-duration actions and complex scene transitions, so the existing naive auto-regressive methods for synthesizing long videos cannot generate human skills. To address this, we propose a novel task, the Key-step Skill Generation (KS-Gen), aimed at reducing the complexity of generating human skill videos. Given the initial state and a skill description, the task is to generate video clips of key steps to complete the skill, rather than a full-length video. To support this task, we introduce a carefully curated dataset and define multiple evaluation metrics to assess performance. Considering the complexity of KS-Gen, we propose a new framework for this task. First, a multimodal large language model (MLLM) generates descriptions for key steps using retrieval argument. Subsequently, we use a Key-step Image Generator (KIG) to address the discontinuity between key steps in skill videos. Finally, a video generation model uses these descriptions and key-step images to generate video clips of the key steps with high temporal consistency. We offer a detailed analysis of the results, hoping to provide more insights on human skill generation. All models and data are available at https://github.com/MCG-NJU/KS-Gen.

Figures

Figures reproduced from arXiv: 2502.08234 by the authors.

Figure 1
Figure 1. This figure presents three different tasks related to hu [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of key-step skill generator. Taking the skill “make matcha” as an example, this skill includes three key steps. First, based on the given initial image and skill description, we generate detailed descriptions of the three steps through a MLLM using retrieval argument (RAG). (The figure shows the simplified step descriptions.) Then, We input the initial image and step description into the Key-step Image Gene… view at source ↗
Figure 3
Figure 3. Key-step Image Generation. The input consists of an initial image and step descriptions, from which features are ex￾tracted using the IP-Adapter image encoder and CLIP text encoder, respectively. These image and text features are fed into a multi￾layer Transformer decoder to autoregressively generate the image features for subsequent clips. The predicted features are then in￾jected into Stable Diffusion XL with IP-A… view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: The visulization with different image generation models. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The visualization with different data curation. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 4
Figure 4. Figure 4: The visualization of the skill generator. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 8
Figure 8. Figure 8: The template of optical flow histogram magnitude histograms are then generated, with the mag￾nitude range on a log scale from 2 −7 to 2 5 , divided into 256 bins. For each optical flow histogram, we compute the KL divergence from a predefined histogram template. This t…
Figure 9
Figure 9. Figure 9: Correlation between automatic metrics and human eval [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: The visualization of the skill generator. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: An example of an open-set complex skill. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 37 canonical work pages

  1. [1]

    Ht-step: Aligning instructional articles with how-to videos.Advances in Neural Information Processing Systems, 36, 2024

    Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagara- jan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to videos.Advances in Neural Information Processing Systems, 36, 2024. 2, 3, 9

    work page 2024

  2. [2]

    Character region awareness for text detec- tion

    Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. Character region awareness for text detec- tion. In CVPR, pages 9365–9374. Computer Vision Founda- tion / IEEE, 2019. 3, 9

    work page 2019

  3. [3]

    Gen2act: Hu- man video generation in novel scenarios enables generaliz- able robot manipulation

    Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Hu- man video generation in novel scenarios enables generaliz- able robot manipulation. arXiv preprint arXiv:2409.16283,

    arXiv

  4. [4]

    Stable video diffusion: Scal- ing latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jam- pani, and Robin Rombach. Stable video diffusion: Scal- ing latent video diffusion models to large datasets. CoRR, abs/2311.15127, 2023. 1, 2, 3, 5, 6, 7, 12

    arXiv 2023

  5. [5]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, et al. Video generation models as world simulators. 2024. URL https://openai. com/research/video- generation-models-as-world-simulators, 3, 2024. 3

    2024

  6. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- mix with water for creating ink grind roundly and evenly Put On Hair Extension dig a pit with proper size fill the tree into the pit fill the pit with some soil Plant Tree put ...

    work page

  7. [7]

    Procedure planning in in- 12 structional videos

    Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in in- 12 structional videos. In ECCV (11), pages 334–350. Springer,

    work page

  8. [8]

    V AST: A vision-audio-subtitle-text omni-modality foundation model and dataset

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. V AST: A vision-audio-subtitle-text omni-modality foundation model and dataset. In NeurIPS, 2023. 3, 9

    work page 2023

Show all 60 references
  1. [9]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In Pro- ceedings of the IEEE/CVF Conference on Computer ...

    2023

  2. [10]

    Claude 3.5 sonnet

    Claude. Claude 3.5 sonnet. 2024. 4, 7

    2024

  3. [11]

    Animateanything: Fine- grained open domain image animation with motion guid- ance

    Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animateanything: Fine- grained open domain image animation with motion guid- ance. arXiv e-prints, pages arXiv–2311, 2023. 1, 2, 3, 5, 6, 7, 12

    2023

  4. [12]

    The EPIC-KITCHENS dataset: Collection, challenges and baselines

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The EPIC-KITCHENS dataset: Collection, challenges and baselines. IEEE Trans. Pattern Anal. Mac...

    2021

  5. [13]

    Doell, and Jason J

    Pradipto Das, Chenliang Xu, Richard F. Doell, and Jason J. Corso. A thousand frames in just a few words: Lingual de- scription of videos through latent topics and sparse object stitching. In CVPR, pages 2634–2641. IEEE Computer So- ciety, 2013. 2

    2013

  6. [14]

    Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tompson

    Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tompson. Video language planning. CoRR, abs/2310.10625, 2023. 3

    arXiv

  7. [15]

    Learning universal policies via text-guided video generation

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In NeurIPS, 2023. 3

    2023

  8. [16]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  9. [17]

    The ”something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fr¨und, Peter Yianilos, Moritz Mueller- Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The ”something something” video da...

    2017

  10. [18]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. CoRR, abs/2307.04725, 2023. 1, 2, 3

    2023 arXiv

  11. [19]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In NIPS, pages 6626–6637, 2017. 4

    2017

  12. [20]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 3

    2020

  13. [21]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  14. [22]

    The kinetics human action video dataset

    Will Kay, Jo ˜ao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017. 2, 3, 9

    2017 arXiv

  15. [23]

    The language of actions: Recovering the syntax and semantics of goal-directed human activities

    Hilde Kuehne, Ali Bilgin Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, pages 780–787. IEEE Computer Society, 2014. 2

    2014

  16. [24]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023. 3, 9

    2023

  17. [25]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. CoRR, abs/2401.03048, 2024. 1, 2, 3

    2024 arXiv

  18. [26]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, pages 2630–2640. IEEE, 2019. 1, 2

    2019

  19. [27]

    Gpt-4o release

    OpenAI. Gpt-4o release. 2024. 4, 7

    2024

  20. [28]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence

  21. [29]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 10

    2023 arXiv

  22. [30]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  23. [31]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 5, 6, 8, 11

    2023 arXiv

  24. [32]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  25. [33]

    Hierarchical text-conditional image gener- 13 ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- 13 ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 1

    2022 arXiv

  26. [34]

    Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models

    Jens Rasmussen. Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans. Syst. Man Cybern. , 13(3):257–266,

  27. [35]

    A database for fine grained activity detec- tion of cooking activities

    Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele. A database for fine grained activity detec- tion of cooking activities. In CVPR, pages 1194–1201. IEEE Computer Society, 2012. 2

    2012

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  29. [37]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  30. [38]

    As- sembly101: A large-scale multi-view video dataset for un- derstanding procedural activities

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. As- sembly101: A large-scale multi-view video dataset for un- derstanding procedural activities. In CVPR, pages 21064– 21074. IEEE, 2022. 2

    2022

  31. [39]

    Tulyakov, and Mohamed Elhoseiny

    Ivan Skorokhodov, S. Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, im- age quality and perks of stylegan2. 2022 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 3616–3626, 2021. 10

    2022

  32. [40]

    Look for the change: Learning object states and state-modifying actions from untrimmed web videos

    Tom ´aˇs Souˇcek, Jean-Baptiste Alayrac, Antoine Miech, Ivan Laptev, and Josef Sivic. Look for the change: Learning object states and state-modifying actions from untrimmed web videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2022

  33. [41]

    Genhowto: Learning to generate actions and state transformations from instructional videos

    Tom ´as Soucek, Dima Damen, Michael Wray, Ivan Laptev, and Josef Sivic. Genhowto: Learning to generate actions and state transformations from instructional videos. CoRR, abs/2312.07322, 2023. 6, 8, 11

    2023 arXiv

  34. [42]

    Sebastian Stein and Stephen J. McKenna. Combining em- bedded accelerometers with computer vision for recogniz- ing food preparation activities. In UbiComp, pages 729–738. ACM, 2013. 2

    2013

  35. [43]

    Plate: Visually-grounded plan- ning with transformers in procedural tasks

    Jiankai Sun, De-An Huang, Bo Lu, Yun-Hui Liu, Bolei Zhou, and Animesh Garg. Plate: Visually-grounded plan- ning with transformers in procedural tasks. IEEE Robotics Autom. Lett., 7(2):4924–4930, 2022. 2

    2022

  36. [44]

    EV A-CLIP: improved training techniques for CLIP at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EV A-CLIP: improved training techniques for CLIP at scale. CoRR, abs/2303.15389, 2023. 3, 9

    2023 arXiv

  37. [45]

    A comprehensive survey of procedural video datasets

    Hui Li Tan, Hongyuan Zhu, Joo-Hwee Lim, and Cheston Tan. A comprehensive survey of procedural video datasets. Comput. Vis. Image Underst., 202:103107, 2021. 1

    2021

  38. [46]

    COIN: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: A large-scale dataset for comprehensive instructional video analysis. In CVPR, pages 1207–1216. Computer Vision Foundation / IEEE, 2019. 1, 2, 3

    2019

  39. [47]

    Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis

    Kolors Team. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. arXiv preprint,

  40. [48]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  41. [49]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022. 4, 10

    2022

  42. [50]

    FVD: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. FVD: A new metric for video generation. In DGS@ICLR. OpenReview.net, 2019. 4

    2019

  43. [51]

    Event-guided procedure planning from in- structional videos with text supervision

    An-Lan Wang, Kun-Yu Lin, Jia-Run Du, Jingke Meng, and Wei-Shi Zheng. Event-guided procedure planning from in- structional videos with text supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 13565–13575, 2023. 2

    2023

  44. [52]

    PDPP: projected diffusion for procedure planning in instructional videos

    Hanlin Wang, Yilu Wu, Sheng Guo, and Limin Wang. PDPP: projected diffusion for procedure planning in instructional videos. In CVPR, pages 14836–14845. IEEE, 2023. 2

    2023

  45. [53]

    Pandora: Towards general world model with natural language actions and video states

    Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.09455, 2024. 2, 3

    2024 arXiv

  46. [54]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Eu- ropean Conference on Computer Vision , pages 399–417. Springer, 202...

    2025

  47. [55]

    Learning interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. CoRR, abs/2310.06114,

    arXiv

  48. [56]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 1, 2, 5, 6, 7, 8, 12

    2024 arXiv

  49. [57]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  50. [58]

    Der- panis, Richard P

    He Zhao, Isma Hadji, Nikita Dvornik, Konstantinos G. Der- panis, Richard P. Wildes, and Allan D. Jepson. P 3iv: Prob- abilistic procedure planning from instructional videos with weak supervision. In CVPR, pages 2928–2938. IEEE, 2022. 2

    2022

  51. [59]

    Luowei Zhou, Chenliang Xu, and Jason J. Corso. To- wards automatic learning of procedures from web instruc- 14 tional videos. InAAAI, pages 7590–7598. AAAI Press, 2018. 2

    2018

  52. [60]

    Fouhey, Ivan Laptev, and Josef Sivic

    Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David F. Fouhey, Ivan Laptev, and Josef Sivic. Cross- task weakly supervised learning from instructional videos. In CVPR, pages 3537–3545. Computer Vision Foundation / IEEE, 2019. 2, 3, 6 15

    2019

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.