{"id":"e24e0336-f2aa-4850-bd19-8e9c148716fe","arxiv_id":"2412.01987","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A video diffusion model generates scene-conditioned, step-by-step visual instructions from an input image and text prompts, trained on a new 0.6M-sequence dataset.","lead":"ShowHowTo generates a sequence of images that visually walk a user through each step of a task, grounded in a photo of their own workspace or ingredients. The paper also introduces a fully automatic pipeline that mined 578K such step-image sequences from 1.2 million instructional videos.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Closed evaluation loop: the SOTA claim in Table 1 is measured with the same DFN-CLIP model used to curate the training set and to filter the test set, so the headline margins may be an artifact of scorer alignment rather than human-judged instruction quality.","rationale":"The reader's conditional verdict is appropriate. I agree with the reader that automatic frame alignment is a real risk: Appendix A's human preference results (36% manual, 18% automatic, 46% tie) are not strong evidence of high alignment quality. However, I see the more load-bearing issue as the closed evaluation loop: the curation scorer (DFN-CLIP, Sec. 3.1), the test-set filter (Sec. 5.1), and two of the three headline metrics (Sec. 5.1) all use the same model. This makes the Table 1 'state-of-the-art on all three dimensions' claim fragile in a way that the reader's stated weakest assumption does not fully capture. The external WikiHow and GenHowTo evaluations only partially break the loop, and the user study, while valuable, measures relative preference rather than absolute correctness. The recommended condition should therefore be explicit: an evaluation variant whose scorer is not DFN-CLIP and whose test set is not filtered by DFN-CLIP alignment, with confidence intervals. This is why agreement_with_reader is 'partial': the reader's rationale does mention the shared-scorer concern, but their weakest_assumption field is about narration-frame alignment, which I consider secondary to the metric circularity.","tokens_in":25751,"tokens_out":7647,"duration_ms":74214,"concrete_test":"Construct a random test set of 3,964 sequences sampled uniformly from the full 578K dataset with no DFN-CLIP alignment filter. Generate sequences with ShowHowTo and the two strongest conditional baselines (Phung et al. and StackedDiffusion). Score Step and Task Faithfulness twice: (a) with a CLIP variant not used in curation (e.g., SigLIP or EVA-CLIP) and (b) with human raters judging whether each generated image depicts the step text. Report bootstrapped 95% confidence intervals. If ShowHowTo no longer beats all conditional baselines on both scoring variants, the Table 1 SOTA claim is a metric artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ShowHowTo's central claim—state-of-the-art Step, Scene, and Task Faithfulness (Abstract; Table 1)—is vulnerable because the same contrastive model is used at every stage of the loop. In Sec. 3.1, DFN-CLIP selects the representative frame for each LLM-extracted step. In Sec. 5.1, the test set is explicitly built to \"prioritize samples with high DFN-CLIP alignment scores.\" Step Faithfulness and Task Faithfulness are then computed with that same DFN-CLIP model (Sec. 5.1, Appendix D). A model trained to imitate DFN-CLIP-selected pairs is therefore evaluated by the very function that chose its training targets, on a subset chosen to have high values of that function. The user study (Fig. 4) gives independent relative support, but it is pairwise preference, not absolute correctness, and Fig. 12's correlation has only method-level points. The WikiHow zero-shot result partially breaks the curation loop but still uses the same DFN-CLIP scorer. Notably, Table 1 shows Stable Diffusion (no input conditioning) achieves Step Faithfulness 0.70 and \"Copy of the input image\" achieves Scene Consistency 0.62, both above ShowHowTo's 0.52/0.34; the claim stands only by excluding these baselines from the 'visual instruction generation methods' category. Without an evaluation independent of the curation scoring function, the reported SOTA margins may reflect alignment with DFN-CLIP's biases rather than genuine step/scene/task fidelity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ShowHowTo, a method for generating a sequence of step-by-step visual instructions from an input scene image and per-step textual instructions. The authors construct a 578K-sequence dataset (4.5M image-text pairs) automatically from HowTo100M videos by combining WhisperX transcription, Llama-3-based step extraction with timestamps, and DFN-CLIP cross-modal frame selection. They then fine-tune a DynamiCrafter-based latent video diffusion model with per-frame text conditioning and variable-length training, and evaluate on their own test set and WikiHow zero-shot with Step Faithfulness, Scene Consistency, and Task Faithfulness, reporting state-of-the-art results, plus a user study.","tokens_in":26015,"tokens_out":4518,"duration_ms":43756,"significance":"The main contribution is a scalable, fully automatic data-creation pipeline and a public dataset and model release. If the evaluation concerns are addressed, the work is significant because it goes beyond single-step generation or non-contextualized sequences, and it enables scene-grounded multi-step visual instructions for downstream assistive and robotics applications. The authors also provide ablations, a zero-shot GenHowTo evaluation, and public code and model weights, which are useful for reproducibility.","major_comments":[{"comment":"The Step Faithfulness and Task Faithfulness metrics use DFN-CLIP (DFN5B-CLIP-ViT-H-14-378), which is also used in Sec. 3.1 to select representative frames and in Sec. 5.1 to prioritize test samples with high DFN-CLIP alignment scores. This creates a closed evaluation loop: the model is trained on pairs chosen by DFN-CLIP and scored by the same function on a subset selected for high values of that function. The reported margins may therefore reflect scorer alignment rather than human-judged instruction quality. I recommend adding an evaluation that breaks this loop, for example using a different vision-language model or a human-rated subset on an unfiltered test set, and reporting metrics separately for the curated and uncurated test splits.","section":"Sec. 5.1, Table 1, Appendix D"},{"comment":"The claim of state-of-the-art results on all three metrics is not supported by Table 1 as presented. Stable Diffusion without input conditioning achieves Step Faithfulness 0.70 and Task Faithfulness 0.44 on the ShowHowTo test set, both above ShowHowTo's 0.52 and 0.42, and the 'Copy of the input image' baseline achieves Scene Consistency 0.62, above ShowHowTo's 0.34. The SOTA claim is only valid if these baselines are categorically excluded from 'visual instruction generation methods'; this exclusion needs to be stated and justified in the main text, and the claim should be qualified accordingly.","section":"Abstract, Sec. 5.2, Table 1"},{"comment":"The validation of the frame-selection step reports that human annotators preferred the automatically selected frame in 18% of cases, the manually annotated frame in 36%, and ties in 46%. This is a low rate of clear machine wins and does not strongly support the 'high-quality dataset' claim. Since incorrect training pairs can directly hurt the scene-grounding and step-faithfulness guarantees, the paper should quantify the sensitivity of the final model to alignment noise, for example by training on a subset with human-selected frames or by ablating the temporal expansion epsilon.","section":"Sec. 3.1, Appendix A"},{"comment":"Table 1 reports only point estimates without error bars, confidence intervals, or significance tests. Given that several differences are small (for example, Task Faithfulness is 0.42 for ShowHowTo, 0.42 for StackedDiffusion, and 0.44 for Stable Diffusion), it is unclear which differences are reliable. Please report variance over multiple generated samples or bootstrap intervals, especially for the claimed improvements over the closest baselines.","section":"Table 1, Sec. 5.2"}],"minor_comments":[{"comment":"The user study reports nine participants; please state the number of pairwise comparisons per metric and whether the same participants evaluated all criteria.","section":"Sec. 5.2"},{"comment":"The caption says 'Rows 2 and 3 show the generation of visual instructions for the same recipe but conditioned on different input images,' but the visual layout appears to show row 3 as a different recipe; please check the row numbering.","section":"Figure 1 caption"},{"comment":"The row labeled '<=8 steps, randomly sampled' is described in the text as 'randomly sampled' with temporal ordering preserved; please clarify how this differs from the main variable-length training protocol, since both use random subsequences.","section":"Sec. 5.3, Table 4"},{"comment":"The phrase 'degenerative case with one frame' should be 'degenerate case with one frame'.","section":"Appendix E, Table 8"},{"comment":"The paper mentions that the original HowTo100M transcripts are noisy and uses WhisperX instead; it would be helpful to report the fraction of videos for which the WhisperX transcript was used successfully after filtering, since long-video context limits are mentioned later.","section":"Sec. 3.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read ShowHowTo. It's a serious empirical paper: new task (scene-conditioned multi-step visual instruction generation), a fully automatic pipeline that turns HowTo100M videos into 578K image-text sequences, and a DynamiCrafter-based video diffusion model that generates variable-length instruction sequences while preserving the input scene. The dataset alone is a real resource. The per-step text conditioning and variable-length training ablations are sound, and the qualitative results are genuinely good. Code, data, and weights are public.\n\nThe soft spot is the evaluation. The stress-test note is right: DFN-CLIP picks the representative frame for each step, filters the test set toward high DFN-CLIP alignment, and then computes Step and Task Faithfulness with that same model. So the headline numbers in Table 1 partly measure how well the model reproduces the curation scorer's preferences. The paper also excludes Stable Diffusion and Copy-input from the 'visual instruction generation methods' category even though those baselines beat ShowHowTo on Step Faithfulness (0.70 vs 0.52) and Scene Consistency (0.62 vs 0.34) respectively. That is not a nitpick; the SOTA claim depends on that exclusion.\n\nThat said, the circularity is partial, not total. The user study is independent and pairwise, and ShowHowTo wins most comparisons—including a 42% win rate against ground-truth source sequences on step and scene, which is a strong result. The GenHowTo zero-shot benchmark uses an external protocol and shows a clear improvement over prior work. The frame-selection human study is less reassuring: annotators preferred the automatic frame in only 36% of cases (46% ties, 18% machine wins), so the data pipeline has real noise, though the authors are honest about it in the appendix.\n\nBottom line: the core contributions hold up. The dataset and model are valuable, and the central idea is sound. The SOTA claim is overstated given the closed loop. A serious referee should engage with this paper—the fixes are clear: report error bars, evaluate on a random test subset, and add a non-DFN-CLIP evaluation. If those come back clean, this becomes a strong paper. Recommend conditional acceptance focusing on the evaluation concerns.","headline":"A valuable dataset and model paper whose headline SOTA claim is weakened by a closed evaluation loop—DFN-CLIP curates the data, filters the test set, and scores the results; the user study and external benchmark suggest the approach is real, but the evaluation needs an independent pass.","tokens_in":26606,"tokens_out":3052,"would_cite":true,"duration_ms":27432,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a video diffusion model on 578,000 instruction sequences mined automatically from web how-to videos lets it generate scene-grounded step-by-step visual instructions, with state-of-the-art scores on step, scene, and task…","keywords":["visual instruction generation","video diffusion model","scene-conditioned generation","step-by-step instructions","automatic dataset collection","instructional videos","cross-modal frame alignment","scene consistency"],"falsifier":"Re-score the generated sequences with a text-image embedding model different from the one used both to select training frames and to compute Step and Task Faithfulness; if the reported margin over baselines shrinks or disappears, the headline scores partly reflect a shared embedding between data curation and evaluation. A complementary check is a hand-labeled alignment audit stratified by task category: if automatically chosen frames match human choice far less often in non-cooking categories than the reported 36 percent, then the training pairs for those categories are misaligned and the scene-grounding claim would not hold there.","tokens_in":25525,"feed_emoji":"🖼️","tokens_out":15527,"duration_ms":105225,"temperature":0.7,"pith_summary":"The paper aims to establish that step-by-step visual instructions — an ordered sequence of images showing how to perform each step of a task — can be generated for a user's own environment, and that this can be learned without any manual annotation. The pipeline first mines 578,000 aligned image-text instruction sequences from one million narrated internet how-to videos, automatically transcribing the narration, extracting step instructions with timestamps, and selecting one representative frame per step. The authors then fine-tune a video diffusion model, ShowHowTo, so that, given an input photo and per-step text prompts, it generates a variable-length image sequence that follows the prompts while keeping the scene (tools, ingredients, workspace) from the input photo. The paper reports that this model beats prior visual-instruction generators on all three axes it measures — step faithfulness, scene consistency, and task faithfulness — and that in a user study human evaluators preferred its outputs over the original source-video frames 42 percent of the time on step and scene quality.","feed_headline":"One input photo becomes a scene-consistent step-by-step guide","feed_subtitle":"Trained on 578K instruction sequences mined from web videos, it keeps every step grounded in the user's own scene.","key_machinery":"Two mechanisms carry the argument. The first is the automatic data-collection pipeline: WhisperX produces accurately time-stamped transcripts of the narration, Llama 3 filters out non-instructional videos and extracts concise step instructions with start and end timestamps, and DFN-CLIP selects the single representative frame for each step inside the timestamp interval expanded by $\\epsilon = 15$ seconds, using dynamic programming so the chosen frames respect the step ordering. The second is the ShowHowTo generation model, a latent video diffusion U-Net initialized from the DynamiCrafter image-animation checkpoint: the input image is encoded and concatenated to the noise latents of every frame, each frame attends to its own step prompt through cross-attention, and temporal attention across the whole sequence keeps objects and background consistent. Training on variable-length consecutive subsequences (two to eight steps) is what converts a fixed-length video model into a generator of sparse, variable-length instruction sequences, and the ablations show that per-frame text conditioning and consecutive-subsequence training are each load-bearing.","core_discovery":"On its own terms, the paper claims that scene-conditioned multi-step instruction generation is a solvable problem, given enough aligned training data and the right conditioning scheme, and that both ingredients can be obtained automatically. Narrated web videos can be converted into a large instruction dataset (578K sequences spanning 25,026 tasks and 4.5M image-text pairs), and a latent video diffusion model fine-tuned on those sequences learns to generate new instruction sequences that respect both the per-step text and the provided scene image. The authors show that per-frame text conditioning is essential — collapsing all steps into a single prompt drops step faithfulness from 0.52 to below 0.30 — and that training on variable-length consecutive subsequences is what allows the model to output sequences of 1 to 15 steps at inference time. They further report that the generated images score higher on step faithfulness than the original dataset frames, because real video frames sometimes fail to show the narrated action clearly.","pith_inferences":["The evaluation and the training data share the same embedding model: DFN-CLIP both selects the training frames and scores Step and Task Faithfulness, so re-scoring with an independent retriever would test whether the reported margins reflect genuine faithfulness or a shared embedding bias — a check the paper does not perform.","The 36 percent human-preference figure for automatically selected frames (46 percent ties, 18 percent machine wins) comes from a small hand-labeled sample, so a stratified audit across task categories could reveal whether narration-visual alignment, the pipeline's weakest link, degrades for rare or non-cooking tasks.","The Scene Consistency metric retrieves from the same source video, which rewards matching the exact recording rather than the functional scene; a harder test would ask whether a person or a robot policy can actually complete the task by following the generated images.","The comparison set suggests that scene grounding and step faithfulness are not an inherent trade-off — joint temporal attention obtains both, where single-step editors get grounding but not steps and sequence generators get steps but not grounding — which points toward stronger object-state conditioning as the natural next step."],"forward_implications":["Given a photo of the user's ingredients, tools, and workspace, the model can produce a complete image guide of 1 to 15 steps that keeps the same countertop, utensils, and objects across the whole sequence, so the guidance is tied to the user's actual environment rather than a generic kitchen or workshop.","The automatic pipeline removes the manual-annotation bottleneck: the same recipe can be re-run on new video corpora to extend instruction generation to novel tasks and languages without any human labeling.","Per-frame text conditioning is decisive; any design that merges the step prompts into a single conditioning signal loses roughly half the step-faithfulness score, so future instruction generators should attend to each step separately.","Because generated images can beat the source-video frames on step faithfulness, the model can serve as a cleaner source of instruction frames than the videos themselves, which matters when the original footage shows a step poorly or not at all.","The authors note the clearest remaining failure mode: object states can revert across long sequences (raw meat reappearing after a cooking step), so state tracking across steps, not scene preservation, is the bottleneck limiting longer generations."],"supporting_citations":[{"why":"DynamiCrafter, the pretrained latent video diffusion model whose weights initialize ShowHowTo and whose architecture supplies the spatial and temporal attention backbone.","marker":"[56]"},{"why":"HowTo100M, the corpus of 1.2 million narrated instructional videos that the automatic pipeline mines for training sequences.","marker":"[42]"},{"why":"WhisperX, the transcription model whose accurately timestamped sentences are the input for step extraction and alignment.","marker":"[5]"},{"why":"Llama 3, the LLM that filters non-instructional videos and extracts step instructions with start and end timestamps.","marker":"[19]"},{"why":"DFN-CLIP, the contrastive model that selects representative frames for each step and later computes the Step and Task Faithfulness metrics.","marker":"[21]"},{"why":"DINOv2, the visual feature extractor used to measure Scene Consistency by retrieval against source-video frames.","marker":"[44]"},{"why":"StackedDiffusion, the main sequence-generation baseline the method must beat, and the source of the metric design the paper adapts.","marker":"[41]"},{"why":"HowToStep, the automatically built cooking dataset used in the training-data ablation to show the superiority of the paper's step extraction.","marker":"[37]"},{"why":"WikiHow-VGSI, the illustrated WikiHow dataset used as a zero-shot test set and as a training-data comparison.","marker":"[62]"},{"why":"GenHowTo, the prior visual-instruction method that supplies a comparison baseline and the zero-shot action-accuracy benchmark.","marker":"[53]"}],"fun_headline_variants":["A single photo becomes a grounded, step-by-step visual how-to","Scene-grounded visual instructions generated from one input image","Diffusion model makes step-by-step visual guides from a photo","Turn one photo into a scene-consistent how-to sequence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach rests on the assumption that what the narrator says is happening on screen at roughly the same time, so the timestamps extracted from the narration and the CLIP-scored frames pick the picture that truly shows each step; the authors' own small human study found the automatically chosen frame was preferred over a manually chosen one only 36 percent of the time.","fun_headline_variants_meta":{"raw":{"variants":["A single photo becomes a grounded, step-by-step visual how-to","Scene-grounded visual instructions generated from one input image","Diffusion model makes step-by-step visual guides from a photo","Turn one photo into a scene-consistent how-to sequence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2892,"prompt_tokens":931,"completion_tokens":1961,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1892}},"tokens_in":547,"tokens_out":1961,"duration_ms":154709,"temperature":1.0,"reasoning_tokens":1892,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:56:25.219184+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the generated sequences with a text-image embedding model different from the one used both to select training frames and to compute Step and Task Faithfulness; if the reported margin over baselines shrinks or disappears, the headline scores partly reflect a shared embedding between data curation and evaluation. A complementary check is a hand-labeled alignment audit stratified by task category: if automatically chosen frames match human choice far less often in non-cooking categories than the reported 36 percent, then the training pairs for those categories are misaligned and the scene-grounding claim would not hold there.","supporting_citations":[{"cited_title":"Howto100m: Learning a text-video embedding by watching hundred million narrated video clips","cited_arxiv_id":null,"evidence_quote":"HowTo100M, the corpus of 1.2 million narrated instructional videos that the automatic pipeline mines for training sequences."},{"cited_title":"Whisperx: Time-accurate speech transcription of long- form audio","cited_arxiv_id":null,"evidence_quote":"WhisperX, the transcription model whose accurately timestamped sentences are the input for step extraction and alignment."},{"cited_title":"Dinov2: Learning robust visual features without su- pervision","cited_arxiv_id":null,"evidence_quote":"DINOv2, the visual feature extractor used to measure Scene Consistency by retrieval against source-video frames."},{"cited_title":"Generating illustrated instructions","cited_arxiv_id":null,"evidence_quote":"StackedDiffusion, the main sequence-generation baseline the method must beat, and the source of the metric design the paper adapts."},{"cited_title":"Multi-sentence grounding for long- term instructional video","cited_arxiv_id":null,"evidence_quote":"HowToStep, the automatically built cooking dataset used in the training-data ablation to show the superiority of the paper's step extraction."},{"cited_title":"Visual goal-step inference using wikihow","cited_arxiv_id":null,"evidence_quote":"WikiHow-VGSI, the illustrated WikiHow dataset used as a zero-shot test set and as a training-data comparison."},{"cited_title":"Genhowto: Learning to generate actions and state transformations from instructional videos","cited_arxiv_id":null,"evidence_quote":"GenHowTo, the prior visual-instruction method that supplies a comparison baseline and the zero-shot action-accuracy benchmark."}],"review_version":1}