{"id":"d2664801-bbbd-4a03-a1e8-002d51ed296c","arxiv_id":"2501.12231","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"InsTALL feeds a procedural graph mined from training videos into a multimodal LLM, improving task recognition, action recognition, next-action and plan prediction, and error detection on instructional videos.","lead":"InsTALL is an AI assistant that watches a video or screen recording of a person doing a multi-step task and answers questions about what task is being done, what step is happening, and what should happen next. It adds a task graph learned from training videos to a multimodal language model, and the authors report accuracy gains across five understanding tasks and two new error-detection tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Graph-injection gains may hinge on closed-set action coverage and the accuracy of the online path bGt; the paper's own Future Works admits errors compound, yet no experiment isolates this failure mode.","rationale":"The reader's weakest assumption — that G is faithful and comprehensive — is exactly where the central claim is least secure. The paper's own limitation statement supports this, and the evaluation protocol does not stress-test it because both datasets are closed-set with full action overlap. My proposed tests would settle whether the graph provides a robust inductive bias or merely a closed-set crutch. I agree with the reader's conditional verdict: the method is coherent and the ablations in Table 8 show a real effect, but the generalization and robustness questions remain open.","tokens_in":18297,"tokens_out":5188,"duration_ms":56550,"concrete_test":"Using the existing COIN test set, stratify videos by whether the model's AR prediction for the current clip maps to the correct graph node. Compare AP and PP accuracy between VQG and VQ on the subset where bGt contains an error. If VQG is not better than VQ on that subset, the graph path actively compounds errors. Additionally, run inference with bGt built from ground-truth action labels; if this oracle-path version substantially exceeds the predicted-path version, the method's gains are bottlenecked by the AR-to-graph mapping, not by the graph itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the procedural graph G, mined from training annotations (Alg. 1), is a faithful and complete model of action dependencies for test videos, and that the online search path bGt (Eq. 7) constructed by mapping the model's free-form AR text onto G is accurate enough to condition subsequent predictions. Both links are fragile. If the AR output is wrong or the action is absent from VG, bGt contains an erroneous or missing node, and every later prediction is conditioned on that corrupted path. The authors acknowledge this in Future Works: 'errors compound across prediction steps, limiting their efficacy on plan prediction.' The reported experiments, however, only use COIN and CrossTask, whose test sets share the same closed taxonomy and action vocabulary as the training set. In a realistic assistance setting, users perform actions not in the mining set, and the graph would provide misleading context. Moreover, the two novel error-detection tasks are constructed by corrupting videos with actions drawn from the training graph's vocabulary, so the model can succeed by checking graph membership rather than by visual understanding. No ablation separates the benefit of the graph as a generalizable prior from its benefit as a closed-set lookup.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents InsTALL, a multi-modal LLM for real-time assistance on instructional videos. It combines a VideoLLM-style architecture (CLIP encoder, temporal pooling, MLP, Mistral-7B-Instruct with LoRA) with a procedural graph G that is automatically mined from training videos. G is verbalized and injected into the LLM prompt at training and inference time; at inference, the model's recognized actions are mapped onto G to form an online search path bG_t that conditions all subsequent predictions. The model is trained on five sub-tasks—task recognition, action recognition, next-action prediction, plan prediction with and without the goal—and two auxiliary error-detection tasks introduced by corrupting video steps. Experiments on COIN and CrossTask report state-of-the-art results, including large gains when the graph is added to multiple MLLM backbones (VideoLLM-online+, GPT-4o-mini, GPT-4-turbo, GPT-4o).","tokens_in":18510,"tokens_out":5895,"duration_ms":54350,"significance":"If the reported effects are stable, the work makes a useful contribution: it shows a data-driven procedural graph can be injected into MLLMs as an inductive bias, consistently improving several backbones (Table 8), a finding that generalizes the idea of GraphRAG to multimodal instruction following. The algorithmic specification of graph construction (Alg. 1) and online path formation (Alg. 2) is clear and reproducible. The two error-detection tasks are a practical addition for assistive systems. However, the strength of the evidence is tempered by the absence of statistical significance reporting, by the fact that the error tasks are constructed from the same graph used for prediction, and by the lack of robustness analysis for the online path mapping.","major_comments":[{"comment":"Table 8 reports single-run accuracies for each backbone and condition, with no error bars, confidence intervals, or significance tests. Since some gains are small (e.g., +0.6 to +3.3 for VideoLLM-online+ on several tasks) and MLLM training and few-shot prompting are known to be noisy, the claim of 'unanimous improvement' is not statistically substantiated. Please provide results over multiple seeds with a paired significance test across tasks, or at least report variance, to confirm the gains are not within run-to-run noise.","section":"Section 5.3, Table 8"},{"comment":"The two error-detection tasks are generated by substituting a step v not in N(vt) ∪ vt and by shuffling to a sequence not present in the graph G. Consequently, an oracle that only has access to G can solve these tasks without any video understanding. The VQG vs. VQ gap in Table 7C could therefore reflect graph-lookup rather than visual error detection. Please add an ablation that either withholds the graph at inference (while still training with it) or constructs errors using visually plausible out-of-vocabulary actions, to disentangle the graph's role from genuine visual error understanding.","section":"Section 4.4, Eqs. (8)-(9)"},{"comment":"The model conditions all predictions on the online search path bG_t, which is obtained by one-hot mapping the AR output to graph nodes. If the AR output is incorrect or the action is absent from G, bG_t is corrupted and the error propagates to all subsequent conditioned predictions. The Future Works paragraph admits that such errors compound, but the evaluation never isolates this failure mode. Since COIN and CrossTask test sets share the closed action taxonomy used to build G, the benefit of graph injection under recognition failures or out-of-vocabulary actions is unknown. Add a robustness experiment (e.g., injecting controlled AR errors or holding out a subset of action nodes from G) and report how AR, AP, PP, and PP+ degrade relative to the VQ baseline.","section":"Section 4.2, Eq. (7) and Future Works"}],"minor_comments":[{"comment":"The abstract in the main text states that the graph is leveraged 'at inference time,' while the abstract in the submission header and Section 4.3 describe using the graph at both training and inference time. The text should be made consistent, since training-time graph verbalization is explicitly described.","section":"Abstract vs. Section 4.3"},{"comment":"The 'Pooling' row in Table 4 lists the shape as 'T×N×D/W'; it is unclear what W denotes and how spatial pooling is implemented. Please specify the exact tensor transformation.","section":"Table 4"},{"comment":"Equation (1) writes 'min EV,Y' without defining the expectation; please clarify over which random variables the expectation is taken and how it is approximated empirically in training.","section":"Eq. (1)"},{"comment":"Table 8 omits the VideoLLM-online (VQ) row on CrossTask, which makes it difficult to compare the magnitude of graph gains across datasets. Please add the row or explain why this baseline is not evaluated on CrossTask.","section":"Table 8"},{"comment":"The node mapping bv_t = arg max 1_VG(at) is ambiguous when at is free-form text; specify the similarity metric used (e.g., cosine similarity between text embeddings) and the tie-breaking procedure for matching to graph nodes.","section":"Section 4.2, Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about graph circularity is legitimate and should be addressed before publication. The paper fits the journal's scope. I would also encourage the editors to ask the authors to commit to releasing code and model checkpoints, as the reproducibility of the multi-backbone comparisons depends on the exact prompt templates and graph-mapping implementation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the core claim holds up—injecting a mined procedural graph into a streaming MLLM improves all four understanding tasks across several backbones, and Table 8 is the best evidence. But the headline \"assistant\" framing oversells the evaluation. The test sets share a closed action vocabulary with the graph, and the two error-detection tasks are built from that same graph, so part of the gain is a lookup, not visual understanding.\n\nWhat's new: the combination of a procedural graph with an online multimodal LLM, plus the graph-generated conversational data and the two error-detection sub-tasks. That combination is not in VideoLLM-online or TaskGraph. The paper also does something useful by showing the graph helps not just their own Mistral model but also VideoLLM-online+ and GPT-4 variants; that consistency is real evidence the mechanism does something beyond one architecture.\n\nWhere it's soft:\n- No error bars, no significance tests, and no code or data release. A 13-point AP gain cannot be assessed properly without variance.\n- The graph is mined from the same training annotations the model learns to predict. COIN and CrossTask test splits share the same task/action taxonomy, so an action outside the mined graph never appears. That makes the assistance claims hard to transfer to realistic open-ended settings. The paper's own Future Works note admits errors compound across prediction steps, but no experiment measures the online path bGt directly or tests out-of-vocabulary actions.\n- The two new error tasks are constructed by corrupting videos with actions drawn from the graph vocabulary. A model could pass by checking membership rather than by actually understanding the visual mistake. The order-detection version is less vulnerable, but the action-error version needs an ablation that removes the graph membership cue.\n- The graph construction itself is close to TaskGraph's; the novelty is in the conditioning, not in the mining. That's fine, but the paper should say it more plainly.\n\nThe central argument—explicit graph structure reduces the reasoning burden on the LLM—holds up as far as the data go. The limitations are addressable and mostly about evaluation breadth, not about a broken method.\n\nWho this is for: anyone working on instructional-video understanding or video-conditioned LLMs. It deserves a serious referee. I'd ask for significance tests, code/data release, an out-of-vocabulary or cross-dataset transfer test, and an ablation that separates graph-as-prior from graph-as-lookup. With those, it would be a solid accept; without them, it's a conditional.","headline":"Core graph-injection claim holds up across backbones, but the closed-set evaluation and graph-built error tasks mean part of the gain is lookup, not understanding; still worth a serious review.","tokens_in":19055,"tokens_out":2717,"would_cite":true,"duration_ms":29325,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Injecting a procedural graph—automatically mined from training videos—into a multimodal large language model improves its real-time understanding of instructional tasks across recognition, prediction, and error-detection benchmarks.","keywords":["instructional video understanding","multimodal large language models","procedural graph","task recognition","action recognition","plan prediction","error detection","online assistance"],"falsifier":"Take a held-out set of instructional videos whose step sequences deliberately include transitions absent from the training-derived graph, and compare the graph-augmented model against the video-plus-query-only model on action prediction and plan prediction. If the graph-augmented model does not beat the base model on these out-of-graph videos—or performs worse—then the proposed benefit depends on closed-world coverage rather than on the graph representation itself.","tokens_in":18093,"feed_emoji":"🎬","tokens_out":5813,"duration_ms":53063,"temperature":0.7,"pith_summary":"This paper claims that a multimodal large language model gives better real-time assistance on instructional videos when it is given an explicit procedural graph—a directed graph of task steps and their temporal order—mined automatically from training videos—as an extra input alongside video frames and the user query. The claim matters because assistants that watch a screen share or video stream need to know not only what action is happening but what comes next, what the overall task is, and when something has gone wrong; those abilities are exactly what the graph is meant to supply. Across task recognition, action recognition, next-action prediction, plan prediction, and two new error-detection tasks, the graph-augmented model outperforms prior video-understanding methods and closed-source LLM baselines.","feed_headline":"Injected task graphs lift video AI assistants on every tested task","feed_subtitle":"Mining step-dependency graphs from videos and feeding them to a multimodal LLM beats prior methods on recognition, prediction, and error…","key_machinery":"The central object is the procedural graph, a directed graph whose nodes are action step labels and whose edges are temporally ordered transitions between consecutive steps observed in training videos. The load-bearing operation is the online search path: as the video unfolds, the model's predicted action text is mapped to a node via one-hot similarity, and the resulting prefix path is inserted into the prompt. This path carries the dependency information that the LLM would otherwise have to infer from pixels alone.","core_discovery":"On its own terms, the paper establishes that conditioning the answer distribution on a procedural graph consistently improves a multimodal LLM over conditioning on video and query alone. The graph is constructed before training by scanning training videos, collecting step annotations as nodes, and adding an edge for every temporally adjacent pair of steps. At inference, the model recognizes each action as free-form text, maps it onto the nearest graph node, and accumulates a predicted online path; that path is then verbalized and fed into the LLM along with frames and prompts. The experiments report gains on every tested task on both COIN and CrossTask, including large absolute improvements when the same graph is appended to existing online video LLMs and proprietary API-based models, and the authors take this as evidence that explicit dependency structure eases the reasoning burden on the LLM.","pith_inferences":["The paper's comparison of graph sources suggests a design principle: the best source may depend on whether the task rewards focused local context or broad dependency coverage; this could be tested by varying the retrieval pool size.","The larger gain on wrong-step detection than on order-error detection suggests the graph mainly encodes local adjacency; a testable extension is to add explicit global ordering constraints.","A deployed assistant would need the graph to grow as new tasks appear, since the online path is built from a fixed graph; incremental graph expansion at inference is a natural next step.","The paper's future-work admission that errors compound along the graph path implies that better node-mapping or error-recovery mechanisms, rather than larger LLMs, are the main lever for plan prediction."],"forward_implications":["If the central claim holds, adding a mined procedural graph is a drop-in improvement for any multimodal LLM in this setting; the experiments show gains even without retraining the underlying model.","Explicit graphs reduce the LLM's need to do long-range planning from raw video, so gains should be largest on tasks with longer dependencies, such as plan prediction, and on detection of inserted wrong steps.","The same graph can generate streaming-dialog training data, lowering the annotation effort for multi-turn assistance.","Because the graph is built from training data, deployments whose test procedures overlap with training procedures can benefit without new annotation."],"supporting_citations":[{"why":"Supplies the online video LLM architecture and the streaming setting that InsTALL extends, and serves as the primary baseline.","marker":"[14]"},{"why":"Provides the video-mined task graph idea and a baseline method that also uses graph structure for keystep recognition.","marker":"[7]"},{"why":"COIN dataset, one of the two benchmarks whose videos and step annotations are mined to build the graph and evaluate all tasks.","marker":"[67]"},{"why":"CrossTask dataset, the second benchmark used for graph construction and evaluation.","marker":"[91]"},{"why":"The LLaVA-style architecture (image encoder, MLP connector, LLM) that the model's design follows.","marker":"[47]"},{"why":"Prior result showing LLMs cannot faithfully follow graph dependencies; cited in Future Works to explain compounding errors on plan prediction.","marker":"[58]"},{"why":"wikiHow is used as an alternative text source for graph construction in the ablation study.","marker":"[35]"},{"why":"LoRA provides the efficient fine-tuning method used to train the LLM on the multi-task objectives.","marker":"[29]"}],"fun_headline_variants":["Task graphs boost video AI assistants on all tested tasks","Graph-injected multimodal LLM beats prior video assistants","Injecting procedural graphs sharpens video assistant accuracy","InsTALL: graph-aware multimodal help for real-time task queries","Video AI with task graphs outdoes baselines on every test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The procedural graph is built only from training videos, so the entire method assumes that every action and transition the assistant will see at test time is already represented in the graph; if a test video contains an unseen step or a new ordering, the mapped graph path becomes empty or misleading.","fun_headline_variants_meta":{"raw":{"variants":["Task graphs boost video AI assistants on all tested tasks","Graph-injected multimodal LLM beats prior video assistants","Injecting procedural graphs sharpens video assistant accuracy","InsTALL: graph-aware multimodal help for real-time task queries","Video AI with task graphs outdoes baselines on every test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1215,"prompt_tokens":900,"completion_tokens":315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":235}},"tokens_in":516,"tokens_out":315,"duration_ms":3656,"temperature":1.0,"reasoning_tokens":235,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:22:15.258513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of instructional videos whose step sequences deliberately include transitions absent from the training-derived graph, and compare the graph-augmented model against the video-plus-query-only model on action prediction and plan prediction. If the graph-augmented model does not beat the base model on these out-of-graph videos—or performs worse—then the proposed benefit depends on closed-world coverage rather than on the graph representation itself.","supporting_citations":[{"cited_title":"Videollm-online: Online video large language model for streaming video","cited_arxiv_id":null,"evidence_quote":"Supplies the online video LLM architecture and the streaming setting that InsTALL extends, and serves as the primary baseline."},{"cited_title":"Coin: A large-scale dataset for comprehensive instructional video analysis","cited_arxiv_id":null,"evidence_quote":"COIN dataset, one of the two benchmarks whose videos and step annotations are mined to build the graph and evaluate all tasks."},{"cited_title":"Cross- task weakly supervised learning from instructional videos","cited_arxiv_id":null,"evidence_quote":"CrossTask dataset, the second benchmark used for graph construction and evaluation."},{"cited_title":"FLAP: Flow-adhering planning with constrained decoding in LLMs","cited_arxiv_id":null,"evidence_quote":"Prior result showing LLMs cannot faithfully follow graph dependencies; cited in Future Works to explain compounding errors on plan prediction."},{"cited_title":"LoRA: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA provides the efficient fine-tuning method used to train the LLM on the multi-task objectives."}],"review_version":1}