Pith. sign in

REVIEW 4 major objections 5 minor 71 references

ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a video diffusion model on 578,000 instruction sequences mined automatically from web how-to videos lets it generate scene-grounded step-by-step visual instructions, with state-of-the-art scores on step, scene, and task…

desk verdict A valuable dataset and model paper whose headline SOTA claim is weakened by a closed evaluation loop—DFN-CLIP curates the data, filters the test set, and scores the results; the user study and external benchmark suggest the approach is real, but the evaluation needs an independent pass. read the letter →

arxiv 2412.01987 v2 pith:KJWFJFBN submitted 2024-12-02 cs.CV

Tomáš Souček , Prajwal Gatti , Michael Wray , Ivan Laptev , Dima Damen , Josef Sivic This is my paper · ORCID
classification cs.CV
keywords visualinstructiongenerationvideodiffusionmodelscene-conditionedstep-by-stepinstructionsautomaticdatasetcollectioninstructionalvideoscross-modalframealignmentsceneconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that step-by-step visual instructions — an ordered sequence of images showing how to perform each step of a task — can be generated for a user's own environment, and that this can be learned without any manual annotation. The pipeline first mines 578,000 aligned image-text instruction sequences from one million narrated internet how-to videos, automatically transcribing the narration, extracting step instructions with timestamps, and selecting one representative frame per step. The authors then fine-tune a video diffusion model, ShowHowTo, so that, given an input photo and per-step text prompts, it generates a variable-length image sequence that follows the prompts while keeping the scene (tools, ingredients, workspace) from the input photo. The paper reports that this model beats prior visual-instruction generators on all three axes it measures — step faithfulness, scene consistency, and task faithfulness — and that in a user study human evaluators preferred its outputs over the original source-video frames 42 percent of the time on step and scene quality.

What carries the argument

Two mechanisms carry the argument. The first is the automatic data-collection pipeline: WhisperX produces accurately time-stamped transcripts of the narration, Llama 3 filters out non-instructional videos and extracts concise step instructions with start and end timestamps, and DFN-CLIP selects the single representative frame for each step inside the timestamp interval expanded by $\epsilon = 15$ seconds, using dynamic programming so the chosen frames respect the step ordering. The second is the ShowHowTo generation model, a latent video diffusion U-Net initialized from the DynamiCrafter image-animation checkpoint: the input image is encoded and concatenated to the noise latents of every frame, each frame attends to its own step prompt through cross-attention, and temporal attention across the whole sequence keeps objects and background consistent. Training on variable-length consecutive subsequences (two to eight steps) is what converts a fixed-length video model into a generator of sparse, variable-length instruction sequences, and the ablations show that per-frame text conditioning and consecutive-subsequence training are each load-bearing.

What would settle it

Re-score the generated sequences with a text-image embedding model different from the one used both to select training frames and to compute Step and Task Faithfulness; if the reported margin over baselines shrinks or disappears, the headline scores partly reflect a shared embedding between data curation and evaluation. A complementary check is a hand-labeled alignment audit stratified by task category: if automatically chosen frames match human choice far less often in non-cooking categories than the reported 36 percent, then the training pairs for those categories are misaligned and the scene-grounding claim would not hold there.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that scene-conditioned multi-step instruction generation is a solvable problem, given enough aligned training data and the right conditioning scheme, and that both ingredients can be obtained automatically. Narrated web videos can be converted into a large instruction dataset (578K sequences spanning 25,026 tasks and 4.5M image-text pairs), and a latent video diffusion model fine-tuned on those sequences learns to generate new instruction sequences that respect both the per-step text and the provided scene image. The authors show that per-frame text conditioning is essential — collapsing all steps into a single prompt drops step faithfulness from 0.52 to below 0.30 — and that training on variable-length consecutive subsequences is what allows the model to output sequences of 1 to 15 steps at inference time. They further report that the generated images score higher on step faithfulness than the original dataset frames, because real video frames sometimes fail to show the narrated action clearly.

Load-bearing premise

The whole approach rests on the assumption that what the narrator says is happening on screen at roughly the same time, so the timestamps extracted from the narration and the CLIP-scored frames pick the picture that truly shows each step; the authors' own small human study found the automatically chosen frame was preferred over a manually chosen one only 36 percent of the time.

Editorial extensions

If this is right

  • Given a photo of the user's ingredients, tools, and workspace, the model can produce a complete image guide of 1 to 15 steps that keeps the same countertop, utensils, and objects across the whole sequence, so the guidance is tied to the user's actual environment rather than a generic kitchen or workshop.
  • The automatic pipeline removes the manual-annotation bottleneck: the same recipe can be re-run on new video corpora to extend instruction generation to novel tasks and languages without any human labeling.
  • Per-frame text conditioning is decisive; any design that merges the step prompts into a single conditioning signal loses roughly half the step-faithfulness score, so future instruction generators should attend to each step separately.
  • Because generated images can beat the source-video frames on step faithfulness, the model can serve as a cleaner source of instruction frames than the videos themselves, which matters when the original footage shows a step poorly or not at all.
  • The authors note the clearest remaining failure mode: object states can revert across long sequences (raw meat reappearing after a cooking step), so state tracking across steps, not scene preservation, is the bottleneck limiting longer generations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evaluation and the training data share the same embedding model: DFN-CLIP both selects the training frames and scores Step and Task Faithfulness, so re-scoring with an independent retriever would test whether the reported margins reflect genuine faithfulness or a shared embedding bias — a check the paper does not perform.
  • The 36 percent human-preference figure for automatically selected frames (46 percent ties, 18 percent machine wins) comes from a small hand-labeled sample, so a stratified audit across task categories could reveal whether narration-visual alignment, the pipeline's weakest link, degrades for rare or non-cooking tasks.
  • The Scene Consistency metric retrieves from the same source video, which rewards matching the exact recording rather than the functional scene; a harder test would ask whether a person or a robot policy can actually complete the task by following the generated images.
  • The comparison set suggests that scene grounding and step faithfulness are not an inherent trade-off — joint temporal attention obtains both, where single-step editors get grounding but not steps and sequence generators get steps but not grounding — which points toward stronger object-state conditioning as the natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ShowHowTo, a method for generating a sequence of step-by-step visual instructions from an input scene image and per-step textual instructions. The authors construct a 578K-sequence dataset (4.5M image-text pairs) automatically from HowTo100M videos by combining WhisperX transcription, Llama-3-based step extraction with timestamps, and DFN-CLIP cross-modal frame selection. They then fine-tune a DynamiCrafter-based latent video diffusion model with per-frame text conditioning and variable-length training, and evaluate on their own test set and WikiHow zero-shot with Step Faithfulness, Scene Consistency, and Task Faithfulness, reporting state-of-the-art results, plus a user study.

Significance. The main contribution is a scalable, fully automatic data-creation pipeline and a public dataset and model release. If the evaluation concerns are addressed, the work is significant because it goes beyond single-step generation or non-contextualized sequences, and it enables scene-grounded multi-step visual instructions for downstream assistive and robotics applications. The authors also provide ablations, a zero-shot GenHowTo evaluation, and public code and model weights, which are useful for reproducibility.

major comments (4)
  1. [Sec. 5.1, Table 1, Appendix D] The Step Faithfulness and Task Faithfulness metrics use DFN-CLIP (DFN5B-CLIP-ViT-H-14-378), which is also used in Sec. 3.1 to select representative frames and in Sec. 5.1 to prioritize test samples with high DFN-CLIP alignment scores. This creates a closed evaluation loop: the model is trained on pairs chosen by DFN-CLIP and scored by the same function on a subset selected for high values of that function. The reported margins may therefore reflect scorer alignment rather than human-judged instruction quality. I recommend adding an evaluation that breaks this loop, for example using a different vision-language model or a human-rated subset on an unfiltered test set, and reporting metrics separately for the curated and uncurated test splits.
  2. [Abstract, Sec. 5.2, Table 1] The claim of state-of-the-art results on all three metrics is not supported by Table 1 as presented. Stable Diffusion without input conditioning achieves Step Faithfulness 0.70 and Task Faithfulness 0.44 on the ShowHowTo test set, both above ShowHowTo's 0.52 and 0.42, and the 'Copy of the input image' baseline achieves Scene Consistency 0.62, above ShowHowTo's 0.34. The SOTA claim is only valid if these baselines are categorically excluded from 'visual instruction generation methods'; this exclusion needs to be stated and justified in the main text, and the claim should be qualified accordingly.
  3. [Sec. 3.1, Appendix A] The validation of the frame-selection step reports that human annotators preferred the automatically selected frame in 18% of cases, the manually annotated frame in 36%, and ties in 46%. This is a low rate of clear machine wins and does not strongly support the 'high-quality dataset' claim. Since incorrect training pairs can directly hurt the scene-grounding and step-faithfulness guarantees, the paper should quantify the sensitivity of the final model to alignment noise, for example by training on a subset with human-selected frames or by ablating the temporal expansion epsilon.
  4. [Table 1, Sec. 5.2] Table 1 reports only point estimates without error bars, confidence intervals, or significance tests. Given that several differences are small (for example, Task Faithfulness is 0.42 for ShowHowTo, 0.42 for StackedDiffusion, and 0.44 for Stable Diffusion), it is unclear which differences are reliable. Please report variance over multiple generated samples or bootstrap intervals, especially for the claimed improvements over the closest baselines.
minor comments (5)
  1. [Sec. 5.2] The user study reports nine participants; please state the number of pairwise comparisons per metric and whether the same participants evaluated all criteria.
  2. [Figure 1 caption] The caption says 'Rows 2 and 3 show the generation of visual instructions for the same recipe but conditioned on different input images,' but the visual layout appears to show row 3 as a different recipe; please check the row numbering.
  3. [Sec. 5.3, Table 4] The row labeled '<=8 steps, randomly sampled' is described in the text as 'randomly sampled' with temporal ordering preserved; please clarify how this differs from the main variable-length training protocol, since both use random subsequences.
  4. [Appendix E, Table 8] The phrase 'degenerative case with one frame' should be 'degenerate case with one frame'.
  5. [Sec. 3.1] The paper mentions that the original HowTo100M transcripts are noisy and uses WhisperX instead; it would be helpful to report the fraction of videos for which the WhisperX transcript was used successfully after filtering, since long-video context limits are mentioned later.

Circularity Check

1 steps flagged · score 4.0 of 10

Closed evaluation loop: DFN-CLIP selects the training frames, filters the test set, and then serves as the Step/Task Faithfulness scorer, so the Table 1 SOTA margins are partly forced by the curation function.

  1. other [Sec. 3.1 (Cross-modal frame alignment); Sec. 5.1 (Dataset; Evaluation metrics); Appendix D]
    "Given these expanded intervals, we compute text-frame similarity scores using DFN-CLIP [21] and select the best alignment that satisfies the temporal ordering of the steps. ... To ensure sample quality, we prioritize samples with high DFN-CLIP alignment scores as measured in our dataset creation pipeline (Section 3). ... Step Faithfulness [41] measures whether each generated image ˆIi correctly depicts its corresponding text instruction τi."

    The same contrastive model is load-bearing at every stage: it chooses the ground-truth frame for each step text (Sec. 3.1), it is used to filter the test set so that only high-scoring DFN-CLIP samples remain (Sec. 5.1), and it is then the scorer for Step Faithfulness and Task Faithfulness (Sec. 5.1, App. D). A model trained to imitate DFN-CLIP-selected image-text pairs is therefore evaluated by the function that generated its supervision, on a test distribution explicitly enriched for that function. The headline SOTA margins in Table 1 are thus partly forced by scorer alignment rather than by independent human-judged fidelity. The loop is not total because the user study and the WikiHow/GenHowTo zero-shot evaluations provide independent evidence, so this is a partial circularity.

full rationale

The central claim of state-of-the-art Step, Scene, and Task Faithfulness is not purely definitional: the model is trained with a generative objective on mined pairs, and the evaluation is an external classification of held-out generated images. However, the same DFN-CLIP model is used at three load-bearing points (frame selection, test-set filtering, metric computation), so the Step and Task Faithfulness numbers in Table 1 partly measure agreement with the curation scorer rather than independent semantic correctness. The paper's own user study (Fig. 4), the WikiHow zero-shot evaluation, and the GenHowTo action-accuracy protocol provide independent checks that keep the central claim from collapsing. The GenHowTo reference is prior work by overlapping authors, but it is used as an external benchmark rather than as an unverified uniqueness theorem, so I do not count it as a circular step. Overall, one closed-loop evaluation step, partially mitigated by external benchmarks: score 4.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The dataset construction relies on several domain assumptions about the alignment of narration and visuals in instructional videos, the reliability of LLM step extraction with timestamps, and the suitability of DFN-CLIP for frame selection. These are reasonable but not proven; they are load-bearing for the claimed data quality.

free parameters (1)
  • temporal expansion epsilon = 15 seconds
    Chosen via a small validation set to balance precision and recall of frame alignment in dataset construction (Appendix A). This is a hand-tuned parameter affecting all training data.
assumptions (4)
  • domain assumption Narrations in instructional videos are roughly aligned with the visual demonstration within an epsilon window.
    Used in Section 3.1 to limit frame search to the LLM-extracted temporal interval expanded by 15s; a known phenomenon cited from [25,37].
  • domain assumption Llama 3 can extract temporally ordered, concise instruction steps with accurate timestamps from WhisperX transcripts.
    Used in Section 3.1 step extraction; the quality of the whole dataset depends on this.
  • domain assumption DFN-CLIP is a reliable estimator of text-image alignment for both selecting representative frames and evaluating generated images.
    Used in dataset creation, test-set selection, and Step/Task Faithfulness metrics (Sections 3.1 and 5.1).
  • domain assumption The pretrained DynamiCrafter checkpoint provides a useful prior for image animation that transfers to instructional step sequences.
    Used in Section 4 training; the model is fine-tuned from this checkpoint and inherits its biases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions." pith.science (2026). https://pith.science/paper/KJWFJFBN

@misc{pith2026241201987,
  author       = {Pith},
  title        = {Pith review of: ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJWFJFBN}},
  note         = {Machine review of arXiv:2412.01987}
}
read the original abstract

The goal of this work is to generate step-by-step visual instructions in the form of a sequence of images, given an input image that provides the scene context and the sequence of textual instructions. This is a challenging problem as it requires generating multi-step image sequences to achieve a complex goal while being grounded in a specific environment. Part of the challenge stems from the lack of large-scale training data for this problem. The contribution of this work is thus three-fold. First, we introduce an automatic approach for collecting large step-by-step visual instruction training data from instructional videos. We apply this approach to one million videos and create a large-scale, high-quality dataset of 0.6M sequences of image-text pairs. Second, we develop and train ShowHowTo, a video diffusion model capable of generating step-by-step visual instructions consistent with the provided input image. Third, we evaluate the generated image sequences across three dimensions of accuracy (step, scene, and task) and show our model achieves state-of-the-art results on all of them. Our code, dataset, and trained models are publicly available.

Figures

Figures reproduced from arXiv: 2412.01987 by the authors.

Figure 1
Figure 1. Given an input image (left) and ordered step-by-step textual instructions for a task (top), [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our automatic approach for creating the ShowHowTo dataset—a large-scale instructional dataset consisting of step-by￾step instruction sequences of image-text pairs to perform diverse HowTo tasks. Examples of step-by-step textual instructions and the corresponding frames are highlighted in green. 3. Building Large-Scale ShowHowTo Dataset Learning to generate visual instructions requires a large￾scale dataset that capt… view at source ↗
Figure 3
Figure 3. Model architecture. Given an input frame I0 (left) and a variable number of text instructions τi describing each step, our diffusion model generates visual instructions Iˆi that correctly follow the prompts τi and are consistent with the input image I0. intended task T in any number of steps 1 ≤ n ≤ 15. We build on recent progress in diffusion models for video generation [56]. However, there are the technical challe… view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: User study results. Win rates of the ShowHowTo method against baselines from pairwise forced decision user eval￾uations, divided into step, scene, and task. Values larger than 50% indicate ShowHowTo is preferred over the other methods (right). Lastly, we show (g) Rando…
Figure 5
Figure 5. Figure 5: Qualitative comparison using the input image (left) and the textual instructions (top) for the task of making a calzone. The images from the source video are shown in the first row. Except for ShowHowTo, methods either struggle to preserve the input scene or to produce…
Figure 6
Figure 6. Figure 6: Qualitative results of our method for sequences from the test set. Given the input image (left) and the textual instructions (top), ShowHowTo generates step-by-step visual instructions while maintaining objects from the input image (e.g., the cooking pot and the cerami…
Figure 7
Figure 7. Figure 7: Frame matching comparison across CLIP, SigLIP, and DFN-CLIP (used in our work). For each method, the figure shows the best matching frame to the instructional text (shown below). instructional steps from video narrations, including the few￾shot examples used (truncated…
Figure 8
Figure 8. Figure 8: Samples from the ShowHowTo dataset. Each sample (row) is a sequence of textual instructions (top) and the associated visual instruction images (bottom). 18.05 - 19.53 ✓ Put aluminum foil in the pressure cooker. 28.86 - 29.70 ✓ Create two little rings using the aluminum…
Figure 9
Figure 9. Figure 9: Comparison between textual instructions extracted by our method (left) and the textual instructions from HowToStep [37] (right) for the same randomly chosen ‘How to bake a potato in the pressure cooker’ video. The original transcript used to produce our instructions is…
Figure 10
Figure 10. Figure 10: Statistics of the ShowHowTo dataset. Left: Distribution of task categories in ShowHowTo dataset. Top-right: Distribution of the number of steps per sequence. Bottom-right: Distribution of the number of words per sentence [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Word cloud visualizations of verbs (left) and nouns (right) in the textual instructions of ShowHowTo dataset. scale. On the other hand, HowToStep [37], similarly to us, leverages HowTo100M videos, but its focus is solely on the cooking domain, and it contains only tem…
Figure 12
Figure 12. Figure 12: Correlation of our metrics with human preference. User study results (Win Rate %) highly correlate with our three metrics—Step Faithfulness, Scene Consistency, and Task Faith￾fulness. D. Evaluation Details As described in Section 5.1, we evaluate our method us￾ing thr…
Figure 13
Figure 13. Figure 13: Qualitative results of our method for sequences from the test set. Our model can generate both short instructional sequences (as shown here) as well as long or very long sequences shown in Figures 14 and 15. Number of generated frames 1 2 3 4 5 6 7 Step Faith. 1.00 0.…
Figure 14
Figure 14. Figure 14: Additional qualitative results of our method for sequences from the test set. Given the input image (left) and the textual instructions (top), ShowHowTo generates step-by-step visual instructions while maintaining objects from the input image. cording to the input ima…
Figure 15
Figure 15. Figure 15: Qualitative results of our method for sequences from the test set. Our model can generate long sequences of visual instructions while being consistent with the input image and the text prompts. Input image Scrape excess material off the cylinder head using a razor bla…
Figure 16
Figure 16. Figure 16: Failure modes. The model can struggle with rare objects and tools (left), or it can fail to update object states after state-changing actions (right). 18 [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Additional qualitative comparison using the input image (left) and the textual instructions (top) for the task of making a cardboard flower with glitter. Only ShowHowTo can produce convincing steps while preserving the input scene. 19 [PITH_FULL_IMAGE:figures/full_fi…
Figure 18
Figure 18. Figure 18: Comparison between WhisperX [4] speech transcription (left) and YouTube ASR (right) for the same ‘How to bake a potato in the pressure cooker’ video. In contrast to YouTube ASR, WhisperX can correctly split the narrations into individual sentences. It also makes signi…
Figure 19
Figure 19. Figure 19: Prompt used for filtering non-instructional videos using Llama 3. Transcript excerpts are truncated for clarity, the full prompt is [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: Prompt used for generating instructional steps with start-end timestamps using Llama 3. The prompt is truncated for clarity, the [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 39 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Ht-step: Aligning instructional articles with how-to videos

    Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagara- jan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to videos. NeurIPS, 2024. 2, 15

    work page 2024

  3. [3]

    Introducing computer use, a new claude 3.5 son- net, and claude 3.5 haiku

    Anthropic. Introducing computer use, a new claude 3.5 son- net, and claude 3.5 haiku. https://www.anthropic. com/news/3-5-models-and-computer-use ,

    work page

  4. [4]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021. 4, 12, 20

    work page 2021

  5. [5]

    Whisperx: Time-accurate speech transcription of long- form audio

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisser- man. Whisperx: Time-accurate speech transcription of long- form audio. INTERSPEECH, 2023. 3

    work page 2023

  6. [6]

    Lumiere: A space- time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Her- rmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space- time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024. 2

    arXiv 2024

  7. [7]

    Gen2act: Hu- man video generation in novel scenarios enables generaliz- able robot manipulation

    Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Hu- man video generation in novel scenarios enables generaliz- able robot manipulation. arXiv preprint arXiv:2409.16283,

    arXiv

  8. [8]

    Zero-shot robotic manipulation with pretrained image- editing diffusion models

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image- editing diffusion models. arXiv preprint arXiv:2310.10639,

    arXiv

Show all 71 references
  1. [9]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    2023 arXiv

  2. [10]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In CVPR, 2023. 2

    2023

  3. [11]

    Generating coherent sequences of visual illustrations for real-world manual tasks

    Jo ˜ao Bordalo, Vasco Ramos, Rodrigo Val´erio, Diogo Gl´oria- Silva, Yonatan Bitton, Michal Yarom, Idan Szpektor, and Joao Magalhaes. Generating coherent sequences of visual illustrations for real-world manual tasks. arXiv preprint arXiv:2405.10122, 2024. 2

    2024 arXiv

  4. [12]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In CVPR, 2023. 5, 8, 15

    2023

  5. [13]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  6. [14]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 2

    arXiv

  7. [15]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In CVPR, 2024. 2

    2024

  8. [16]

    Seine: Short-to-long video diffusion model for generative transition and prediction

    Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In ICLR,

  9. [17]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 2023. 2

    2023

  10. [18]

    Learning universal policies via text-guided video generation

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. NeurIPS, 2023. 2

    2023

  11. [19]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  12. [20]

    Step- former: Self-supervised step discovery and localization in instructional videos

    Nikita Dvornik, Isma Hadji, Ran Zhang, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. Step- former: Self-supervised step discovery and localization in instructional videos. In CVPR, 2023. 2 9

    2023

  13. [21]

    Data fil- tering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data fil- tering networks. arXiv preprint arXiv:2309.17425, 2023. 4, 12, 15

    2023 arXiv

  14. [22]

    Preserve your own correlation: A noise prior for video diffusion models

    Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In ICCV, 2023. 2

    2023

  15. [23]

    Emu video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factoriz- ing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023. 2

    2023 arXiv

  16. [24]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2024. 2

    2024 arXiv

  17. [25]

    Temporal alignment networks for long-term video

    Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long-term video. In CVPR, 2022. 3, 4, 12

    2022

  18. [26]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  19. [27]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. NeurIPS, 2020. 2

    2020

  20. [28]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 2

    arXiv

  21. [29]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. NeurIPS, 2022. 2

    2022

  22. [30]

    Make it move: controllable image-to-video generation with text de- scriptions

    Yaosi Hu, Chong Luo, and Zhenzhong Chen. Make it move: controllable image-to-video generation with text de- scriptions. In CVPR, 2022. 2

    2022

  23. [31]

    An edit friendly ddpm noise space: Inversion and manipulations

    Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In CVPR, 2024. 15

    2024

  24. [32]

    Incorporating task progress knowledge for subgoal generation in robotic manipulation through image edits

    Xuhui Kang and Yen-Ling Kuo. Incorporating task progress knowledge for subgoal generation in robotic manipulation through image edits. arXiv preprint arXiv:2410.11013 ,

    arXiv

  25. [33]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022. 2

    2022

  26. [34]

    Videopoet: A large language model for zero-shot video gen- eration

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video gen- eration. arXiv preprint arXiv:2312.14125, 2023. 2

    2023 arXiv

  27. [35]

    Learning action and reasoning-centric image editing from videos and simulations

    Benno Krojer, Dheeraj Vattikonda, Luis Lara, Varun Jam- pani, Eva Portelance, Christopher Pal, and Siva Reddy. Learning action and reasoning-centric image editing from videos and simulations. arXiv preprint arXiv:2407.03471 ,

    arXiv

  28. [36]

    Lego: Learning egocentric action frame generation via visual instruction tuning

    Bolin Lai, Xiaoliang Dai, Lawrence Chen, Guan Pang, James M Rehg, and Miao Liu. Lego: Learning egocentric action frame generation via visual instruction tuning. arXiv preprint arXiv:2312.03849, 2023. 2, 13, 15

    2023 arXiv

  29. [37]

    Multi-sentence grounding for long- term instructional video

    Zeqian Li, Qirui Chen, Tengda Han, Ya Zhang, Yanfeng Wang, and Weidi Xie. Multi-sentence grounding for long- term instructional video. 2024. 2, 3, 6, 12, 13, 14, 15

    2024

  30. [38]

    Dreamitate: Real-world visuomotor policy learn- ing via video generation

    Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sud- hakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl V ondrick. Dreamitate: Real-world visuomotor policy learn- ing via video generation. arXiv preprint arXiv:2406.16862,

    arXiv

  31. [39]

    Learning to ground instructional articles in videos through narrations

    Effrosyni Mavroudi, Triantafyllos Afouras, and Lorenzo Torresani. Learning to ground instructional articles in videos through narrations. In ICCV, 2023. 2

    2023

  32. [40]

    Vidm: Video implicit diffusion models

    Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. In AAAI, 2023. 2

    2023

  33. [41]

    Generating illustrated instructions

    Sachit Menon, Ishan Misra, and Rohit Girdhar. Generating illustrated instructions. In CVPR, 2024. 2, 5, 7, 14, 15, 17

    2024

  34. [42]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019. 3

    2019

  35. [43]

    Visual reinforcement learn- ing with imagined goals

    Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learn- ing with imagined goals. NeurIPS, 2018. 2

    2018

  36. [44]

    Dinov2: Learning robust visual features without su- pervision

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without su- pervision. TMLR, 2024. 5, 15

    2024

  37. [45]

    Coher- ent zero-shot visual instruction generation

    Quynh Phung, Songwei Ge, and Jia-Bin Huang. Coher- ent zero-shot visual instruction generation. arXiv preprint arXiv:2406.04337, 2024. 2, 5, 7, 14, 15, 17

    2024 arXiv

  38. [46]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  39. [47]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 4, 13

    2021

  40. [48]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 5, 6, 15

    2022

  41. [49]

    Gen-3 alpha

    RunwayML. Gen-3 alpha. 2024. 2

    2024

  42. [50]

    Howtocap- tion: Prompting llms to transform video annotations at scale

    Nina Shvetsova, Anna Kukleva, Xudong Hong, Christian Rupprecht, Bernt Schiele, and Hilde Kuehne. Howtocap- tion: Prompting llms to transform video annotations at scale. In ECCV, 2024. 2, 13, 15

    2024

  43. [51]

    Ego4d goal-step: Toward hierarchical understanding of procedural activities

    Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. NeurIPS, 2024. 2, 13, 15 10

    2024

  44. [52]

    Multi-task learning of object states and state-modifying actions from web videos

    Tom ´aˇs Souˇcek, Jean-Baptiste Alayrac, Antoine Miech, Ivan Laptev, and Josef Sivic. Multi-task learning of object states and state-modifying actions from web videos. TPAMI, 2024. 2

    2024

  45. [53]

    Genhowto: Learning to generate actions and state transformations from instructional videos

    Tom ´aˇs Sou ˇcek, Dima Damen, Michael Wray, Ivan Laptev, and Josef Sivic. Genhowto: Learning to generate actions and state transformations from instructional videos. In CVPR,

  46. [54]

    Coin: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, 2019. 2, 13, 15

    2019

  47. [55]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability. NeurIPS, 2024. 2

    2024

  48. [56]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In ECCV,

  49. [57]

    Flow as the cross-domain manipulation interface

    Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gor- don Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. arXiv preprint arXiv:2407.15208, 2024. 2

    2024 arXiv

  50. [58]

    Learn- ing object state changes in videos: An open-world perspec- tive

    Zihui Xue, Kumar Ashutosh, and Kristen Grauman. Learn- ing object state changes in videos: An open-world perspec- tive. In CVPR, 2024. 2

    2024

  51. [59]

    Unloc: A unified framework for video localization tasks

    Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid. Unloc: A unified framework for video localization tasks. In ICCV, 2023. 2

    2023

  52. [60]

    Dif- fusion probabilistic modeling for video generation

    Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Dif- fusion probabilistic modeling for video generation. Entropy,

  53. [61]

    Learning interactive real-world simulators

    Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuur- mans, and Pieter Abbeel. Learning interactive real-world simulators. In ICLR, 2024. 2

    2024

  54. [62]

    Visual goal-step inference using wikihow

    Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, and Chris Callison-Burch. Visual goal-step inference using wikihow. In EMNLP, 2021. 2, 5, 6, 13, 15

    2021

  55. [63]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2

    2024 arXiv

  56. [64]

    Video probabilistic diffusion models in projected latent space

    Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In CVPR, 2023. 2

    2023

  57. [65]

    Scaling robot learn- ing with semantically imagined experience

    Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learn- ing with semantically imagined experience. arXiv preprint arXiv:2302.11550, 2023. 2

    2023 arXiv

  58. [66]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 4, 13

    2023

  59. [67]

    Show-1: Marrying pixel and latent diffusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. IJCV, 2024. 2

    2024

  60. [68]

    I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145, 2023. 2

    2023 arXiv

  61. [69]

    Magicvideo: Efficient video generation with latent diffusion models

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 2

    2022 arXiv

  62. [70]

    Put some aluminum foil in there

    Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross- task weakly supervised learning from instructional videos. In CVPR, 2019. 2, 13, 15 11 Appendix In the appendix, we first provide dataset collection details and sh...

    2019

  63. [2024]

    2, 5, 6, 8, 12, 13, 15, 16

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.