{"id":"6e79db46-0a35-4c4d-8d13-014b0c1fa1f1","arxiv_id":"2412.11621","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"VG-TVP enriches LLM-generated text plans with captions from instructional videos and produces a short video per step; human raters prefer it over text-only baselines.","lead":"The authors combine a language model, a video captioning model, and a text-to-video generator to produce step-by-step instructions that pair text with short videos for everyday tasks. They also introduce a new dataset of daily-life procedural tasks and report human preference results favoring their method over text-only baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'outperforms' claim rests on human-preference percentages from 28 subjects with no significance tests, confidence intervals, or per-cell counts; margins are often small or negative, so the central claim is not statistically established.","rationale":"The reader's weakest_assumption focuses on VLog caption accuracy. That is a genuine mechanism-level risk, and the paper itself admits in the appendix that FoC can misorder information when VLog mislabels objects. However, it is not the most direct threat to the abstract's claim. The central claim is empirical: human raters preferred VG-TVP over unimodal baselines. That evidence base is statistically fragile. With 28 subjects, no significance tests, and no per-cell counts, the observed win percentages could easily be noise. The paper's own discussion of the GPT-3.5 unseen-task results is post hoc and underscores the inconsistency of the preference evidence. A conditional acceptance should require either a re-analysis with confidence intervals or a larger pre-registered study. This does not change the reader's CONDITIONAL verdict, but it refocuses the condition on evaluation reliability rather than VLog quality. I flag the appendix limitation about VLog mislabeling as an additional risk, but the missing statistical foundation is the load-bearing concern for the stated claim.","tokens_in":28089,"tokens_out":7464,"duration_ms":69631,"concrete_test":"Re-analyze the raw preference data from Tables 1-2: for each baseline-aspect cell, compute the Wilson 95% confidence interval for the win proportion and run a binomial test against the null of 50% win (or McNemar if paired), and report the actual number of ratings per cell. If, for the majority of baselines/aspects, the confidence interval includes 50% or the p-value exceeds 0.05, the claim 'outperforms unimodal baselines' is not statistically supported. If raw counts are not available, the authors should release them or run a new evaluation with pre-registered sample sizes and a multilevel model accounting for subject and task random effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims VG-TVP outperforms unimodal baselines on Daily-PP. The only evidence is the Win-Tie-Lose human evaluation in Tables 1-2. The paper reports aggregate percentages but never states how many ratings underlie each cell, nor performs any significance test, confidence interval, or inter-rater agreement measure. There are 28 subjects for 50 seen and 15 unseen tasks; given the 'no subject sees the same task twice' design, each task-baseline comparison may be rated by only one or very few subjects. Several key comparisons are not clearly favorable: versus GPT-3.5 on unseen tasks, VG-TVP loses plan accuracy (26.67% win vs. 40% lose) and temporal coherence (20% vs. 40%); versus Llama2-13B-q4 on seen textual informativeness, win is 40% vs. 36% lose. Even the favorable visual-informativeness scores (e.g., 74% win) could reflect chance without a statistical test. The appendix attempts to explain the GPT-3.5 anomaly post hoc, but that does not address the missing statistics. Thus the load-bearing assumption for the central claim is that these preference differences are real and not noise; the paper provides no evidence for that. The VLog caption limitation the authors self-report is a separate mechanism-level concern: if captions misorder steps, FoC inherits errors, but the claim is about human preference, which could still be positive even with imperfect captions. The absence of statistical support is the more decisive flaw.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VG-TVP, a zero-shot multimodal procedural planning framework that combines an LLM-generated vanilla text plan, VLog video captions fused through a Fusion of Captioning (FoC) module, and ModelScope text-to-video generation to produce paired textual and video step plans. The authors introduce a new dataset, Daily-PP, with 50 seen and 15 unseen daily-life tasks, and evaluate their method with human win/tie/lose comparisons across textual informativeness, visual informativeness, temporal coherence, and plan accuracy, supplemented by BLEU/METEOR scores and an LLM-based evaluation protocol. The central claim is that VG-TVP outperforms unimodal baselines on Daily-PP.","tokens_in":28389,"tokens_out":5835,"duration_ms":56407,"significance":"If the human-preference results held up statistically, the paper would offer a practical zero-shot recipe for multimodal procedural planning, and the released Daily-PP dataset and code would be reusable assets for the community. The FoC idea of aligning and fusing unordered video captions with vanilla text plans is plausible, and the V2T-B/T2V-B pipeline is a reasonable way to couple video understanding with text-to-video generation. However, the evaluation as presented does not yet establish the central claim: the human evaluation tables lack uncertainty quantification, several comparisons favor the baselines, and the acknowledged caption-error/misordering limitation in FoC is never quantified. The contribution is potentially useful but not yet demonstrated at the bar of the stated claim.","major_comments":[{"comment":"The paper's headline claim rests entirely on the win/tie/lose percentages in Tables 1 and 2, but the manuscript reports no significance test, confidence interval, per-cell rating count, or inter-rater agreement measure. With 28 subjects and 50 seen/15 unseen tasks under the stated 'no subject sees the same task twice' protocol, a margin such as 40% vs. 36% could easily be sampling noise; the authors must state how many ratings contribute to each cell and report, for example, exact binomial confidence intervals and a test that the VG-TVP win proportion exceeds the baseline win proportion (or a tie-aware model). Without this, 'outperforms' is not statistically established.","section":"Human Evaluation Metric; Tables 1 and 2"},{"comment":"The abstract's claim that VG-TVP 'outperforms unimodal baselines' overstates what the tables show. In Table 2 (unseen tasks), VG-TVP loses to GPT-3.5 on temporal coherence (20.00 win vs. 40.00 lose) and on plan accuracy (26.67 vs. 40.00), and ties on textual informativeness (80.00). In Table 1, it also loses textual informativeness to Llama2-13B-q4 (40.00 win vs. 36.00 lose). The claim should be scoped to the aspects and baselines where the data support it, or be backed by the statistical analysis requested above.","section":"Abstract; Results, Tables 1 and 2"},{"comment":"The appendix states that FoC 'may also misorder information' when VLog mislabels objects, giving the example of an apple mislabeled as an orange or lemon based on its color. This directly affects temporal coherence and plan accuracy, which are exactly the aspects where VG-TVP already loses to GPT-3.5 on unseen tasks. Because FoC is the mechanism that grounds text plans in visual content, the paper needs at least a quantitative check of caption and step-order accuracy (e.g., agreement with human-annotated step order on a subset) or an ablation with an alternative captioner; otherwise the visual-grounding link in the pipeline remains unverified.","section":"Appendix, The Impact of V2T-B and FoC"},{"comment":"The protocol description says 'No subject was shown the same task twice,' but it is not explained how the 28 subjects were assigned across the 50 seen and 15 unseen tasks and across the eight baselines in Table 1. As a result, the reader cannot tell whether the comparisons in each cell are based on a single subject, a few subjects, or many, and whether the percentages are comparable across rows. The authors should describe the assignment procedure and report the number of comparisons behind each percentage.","section":"Experiments, Human Evaluation Protocol"}],"minor_comments":[{"comment":"The capitalization of FoC is inconsistent: the Ablations paragraph uses 'FOC' instead of 'FoC'; please unify the terminology.","section":"Ablations"},{"comment":"Table 3 is hard to interpret because the header does not clearly separate the BLEU and METEOR columns for the baseline and VG-TVP conditions, and the BLEU values are near zero; please clarify the layout and consider reporting confidence intervals or omitting the metric if it is not intended to support the main claim.","section":"Table 3"},{"comment":"The appendix text says VG-TVP 'consistently outperforms baselines and TIP' on CLIP-based MSS, but Tables 4 and 5 do not include a TIP row and report no error bars or significance information; please add the TIP comparison and uncertainty estimates or temper the claim.","section":"Tables 4 and 5 and surrounding text"},{"comment":"Several task names appear as concatenated strings such as 'How toBakeKofta/MeatballsandPotatoes?' and 'How toCookSpaghetti'; please use spaced and readable task names in figures and text.","section":"Figures 1, 5, 18, 24"}],"recommendation":"major_revision","confidential_remarks":"This is an empirical pipeline paper whose novelty lies mainly in the FoC fusion idea and the Daily-PP dataset. The self-citation to VLog is not problematic in my view, since VLog is a pre-existing tool used as a component. The main issue is that the human-preference evaluation is not statistically supported and the abstract overclaims relative to Table 2. With added per-cell counts, significance tests or intervals, and a scoped claim, the paper could become acceptable; without those, the central 'outperforms' claim is not established. The manuscript fits the general AI/vision audience but needs a rigorous revision before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: VG-TVP is a sensible extension of TIP from text-image to text-video prompting. The new pieces are the FoC caption fusion step and the Daily-PP dataset with seen/unseen splits. If you work on procedural planning or instructional video understanding, the dataset alone is worth a look. The paper is not a breakthrough, but it is a solid engineering contribution with a mostly favorable human preference study.\n\nWhat it does well: The pipeline is clearly described and the components are standard (LLM, VLog captioning, ModelScope T2V). Daily-PP fills a real gap: existing datasets are either not structured around MPP or lack unseen task splits. The human evaluation design (28 subjects, no subject sees the same task twice) is reasonable for a first pass. The CLIP score supplementary analysis adds some quantitative weight. The appendix is candid about VLog's caption errors and their effect on FoC.\n\nSoft spots, in order of severity. First, the central claim 'outperforms unimodal baselines' is not statistically established. Tables 1-2 give percentages but no significance tests, confidence intervals, or number of ratings per cell. Some cells are close or negative: against GPT-3.5 on unseen tasks, VG-TVP loses on plan accuracy (40% lose vs 26.67% win) and temporal coherence (40% vs 20%). The abstract overstates the result. Second, the FoC ablation is purely qualitative (Figures 4 and 16). There is no quantitative measurement of FoC's contribution. Third, no code is released, only a dataset link. That limits reproducibility, though the dataset is the main artifact.\n\nOn the stress-test note: I think the missing statistics is the decisive issue, not the VLog limitation. The paper's own appendix acknowledges the caption misordering, but that is a mechanism-level concern; the human preference claim could still hold even with imperfect captions. The lack of statistical support is what keeps the paper from being decisive.\n\nWho this is for: researchers building multimodal instructional assistants, and anyone who wants a benchmark for text-video procedural plan generation. It deserves a serious referee; the dataset and pipeline are worth engaging with. My recommendation: send it to review, but require the authors to either add significance testing or soften the abstract to 'generally preferred' claims.","headline":"Useful dataset and pipeline, but the headline 'outperforms' claim lacks statistical support.","tokens_in":28906,"tokens_out":1920,"would_cite":true,"duration_ms":17968,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that grounding LLM plans in video captions and regenerating them as video yields multimodal how-to instructions preferred over text-only baselines.","keywords":["multimodal procedural planning","text-to-video generation","instructional videos","large language models","zero-shot prompting","video captioning","Fusion of Captioning","Daily-PP dataset"],"falsifier":"Show the model a task whose instructional videos contain an object the captioner consistently mislabels (the paper's own example is an apple called an orange), and check whether the generated text and video plans misname or misorder that step; if the plans stay correct despite the bad caption, the claimed grounding is not doing the work.","tokens_in":27924,"feed_emoji":"🎬","tokens_out":4945,"duration_ms":42562,"temperature":0.7,"pith_summary":"This paper tries to establish that a large language model can generate better how-to instructions when it is given not only the task name but also captions pulled from instructional videos, and when the resulting text plan is used to drive a text-to-video model. The claim is that this visually grounded text-video plan beats text-only plans from the same LLM family on human-judged informativeness, temporal coherence, and accuracy. The paper introduces a dataset, Daily-PP, with 50 seen and 15 unseen everyday tasks, because existing procedure-planning datasets do not fit multimodal plan generation. If the claim holds, everyday learners would receive paired step-by-step text and short video demonstrations generated automatically and without training a new model.","feed_headline":"Text-plus-video how-to plans beat text-only steps","feed_subtitle":"Zero-shot pipeline grounds LLM planning in video captions to generate paired text and clip instructions for daily tasks.","key_machinery":"The load-bearing component is the Fusion of Captioning (FoC) step, which takes video captions from several instructional videos of the same task and uses an LLM to reorder, merge, and align them into a single ranked caption list that matches the procedural steps. Around it sit two bridges: Video-to-Text (V2T-B), which turns instructional videos into captions, and Text-to-Video (T2V-B), which turns the revised visual descriptions into short clips. FoC is what converts scattered, timestamped captions into a visually grounded text plan. The paper claims that without FoC, the generated videos lose plan accuracy and visual informativeness.","core_discovery":"The central discovery is that video knowledge enters an LLM most usefully as fused caption text rather than as raw pixels, and that the key difficulty is temporal alignment between what the video captions show and what the text plan says. The proposed VG-TVP pipeline first asks an LLM for a vanilla text plan, then captions multiple instructional videos of the same task, uses a Fusion of Captioning step to reorder and merge those captions into a coherent procedure, and then prompts the LLM to rewrite the original plan as text/context/visual triplets. The visual sentence in each triplet is fed to a text-to-video model, producing a short video per step. Human preference experiments on the Daily-PP dataset show the resulting multimodal plans winning against unimodal baselines on most comparisons, with the largest margins in visual informativeness.","pith_inferences":["The quality ceiling of the whole pipeline is set by the captioning step, so a better video-understanding model should directly raise both text and video plan quality without any redesign.","The same caption-fusion recipe could be transferred to other instruction formats, such as diagrams or audio narration, whenever a model can turn those modalities into text.","A testable extension would measure whether users actually complete tasks faster or more successfully with text-video plans than with text-only plans, since the paper evaluates preference, not performance."],"forward_implications":["Procedural planning can be improved without fine-tuning or training new models: composing a zero-shot LLM, a video captioning model, and a text-to-video model is enough.","For tasks that have no instructional videos, the model can still produce multimodal plans by borrowing and fusing captions from related seen tasks.","Generated step videos are short and human-centred, which the paper argues can lower a learner's cognitive load compared with watching long instructional videos.","Human preference, not BLEU or METEOR, is the appropriate yardstick for daily-life procedural plans, because such tasks have no single ground-truth sequence."],"supporting_citations":[{"why":"Frames multimodal procedural planning as dual text-image prompting and supplies the TIP baseline that VG-TVP extends to text-video.","marker":"Lu et al. 2023b"},{"why":"Supplies the zero-shot chain-of-thought prompting used to generate the vanilla text plans.","marker":"Kojima et al. 2022"},{"why":"Establishes LLMs as zero-shot planners, the capability VG-TVP builds on.","marker":"Huang et al. 2022"},{"why":"Provides VLog, the video captioning model used in the Video-to-Text bridge to extract captions from instructional videos.","marker":"Lin and Lei 2023"},{"why":"Provides ModelScope, the text-to-video model used to render each revised visual description as a clip.","marker":"Wang et al. 2023b"},{"why":"Supplies the BLEU metric used to compare generated text plans against reference plans.","marker":"Papineni et al. 2002"},{"why":"Supplies the METEOR metric used alongside BLEU for reference-based text evaluation.","marker":"Banerjee and Lavie 2005"},{"why":"Supplies the Llama2 family of baselines against which VG-TVP is compared in human evaluations.","marker":"Touvron et al. 2023"}],"fun_headline_variants":["Video captions, not pixels, guide LLM how-to plans","Multimodal plans beat text-only when videos are captioned","LLM planning improves with fused video captions","Text-video bridges align plans for daily tasks","Fused captions yield better multimodal how-to plans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the video captioning model describes the real objects and actions accurately enough for the LLM to trust; the appendix notes that when VLog mislabels an object, FoC can propagate the mistake and misorder the plan.","fun_headline_variants_meta":{"raw":{"variants":["Video captions, not pixels, guide LLM how-to plans","Multimodal plans beat text-only when videos are captioned","LLM planning improves with fused video captions","Text-video bridges align plans for daily tasks","Fused captions yield better multimodal how-to plans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1497,"prompt_tokens":985,"completion_tokens":512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":433}},"tokens_in":601,"tokens_out":512,"duration_ms":5221,"temperature":1.0,"reasoning_tokens":433,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:45:25.432743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show the model a task whose instructional videos contain an object the captioner consistently mislabels (the paper's own example is an apple called an orange), and check whether the generated text and video plans misname or misorder that step; if the plans stay correct despite the bad caption, the claimed grounding is not doing the work.","supporting_citations":[{"cited_title":"S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y","cited_arxiv_id":null,"evidence_quote":"Supplies the zero-shot chain-of-thought prompting used to generate the vanilla text plans."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes LLMs as zero-shot planners, the capability VG-TVP builds on."}],"review_version":1}