{"id":"d1e91e44-d013-4965-8b67-a8a59b84071c","arxiv_id":"2506.12623","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MS4UI provides a new benchmark for summarizing UI instructional videos into step-by-step text and key frames, where existing models perform poorly.","lead":"This paper introduces MS4UI, a dataset of 2,413 Adobe user-interface tutorial videos with step segmentation, text instructions, and key frame annotations. It shows that current multi-modal summarization models score low on these tasks, motivating new methods for UI-specific video understanding.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth provenance is unmeasured: GPT-3.5-generated segmentations and summaries pass through human verification with no reported agreement, so the benchmark's validity and headline 'methods struggle' conclusion rest on an unquantified assumption.","rationale":"The reader's weakest assumption was that the GPT-3.5-generated annotations, after human verification, are accurate ground truth. That is exactly the most load-bearing concern in this paper. Every experimental result and the dataset's utility depend on annotation quality, yet the paper provides no measurement of it. The abstract's 'manually annotated' wording is inconsistent with the pipeline in Section 3.2, where GPT-3.5 generates the initial segmentation and summaries and humans verify them. This is not an accusation of bad faith; it is a request for evidence. The proposed re-annotation test would settle whether the concern lands. If human-human agreement is high and the released labels match that agreement, the benchmark's core claims stand and the conditional can be lifted. If not, the paper's conclusions about existing methods struggling on UI video summarization are ungrounded. Other potential issues, such as missing error bars or the details of the Recall@K key-frame metric, are secondary and addressable once the reference labels are shown to be reliable. Because the reader already conditioned acceptance on this issue, my stress-test does not change the verdict; it reinforces it with a concrete way to check the weakest point.","tokens_in":7567,"tokens_out":5028,"duration_ms":58531,"concrete_test":"Take a random sample of 200 videos from MS4UI and re-annotate each with two independent human annotators who see only the videos and the same instructions described in Section 3.2, never the GPT-3.5 outputs. Compute (1) segment-boundary agreement via F1 at a one-second tolerance, (2) pairwise ROUGE-1 and ROUGE-2 between the two annotators' step summaries, and (3) key-frame agreement at one frame per second. Then compare the released GPT-3.5-based labels to this human-human agreement distribution. If the released labels fall outside the human-human agreement band, or if human-human agreement itself is low (for example, boundary F1 below 0.7 or ROUGE-1 below 0.5), the ground truth is too noisy for the baseline comparisons in Tables 2-4 to support the paper's conclusions. If the released labels sit comfortably within the human-human band, the annotation-quality concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MS4UI is the first UI instructional video dataset with segmentation plus multimodal summarization and that existing methods struggle on it—presupposes that the ground-truth annotations are accurate enough to make the baseline numbers meaningful. Section 3.2 states that GPT-3.5 is used to segment and summarize video transcriptions, and that human workers verify and revise the outputs. The paper reports no annotator count, no inter-annotator agreement, no quality-control metrics, and no comparison of the verified labels against independent manual annotations. This matters because the reference labels inherit GPT-3.5's biases: summaries may be transcript-bound, may contain plausible but incorrect UI terms, and segmentation boundaries may systematically drift. The abstract and introduction describe the data as 'manually annotated,' which conflicts with the GPT-3.5-first pipeline and makes the provenance unclear. If the human verification is superficial or inconsistent, the low MIOU, ROUGE, and Recall values in Tables 2-4 measure disagreement with noisy references rather than genuine difficulty of UI video understanding. The Limitations section does not acknowledge this risk, and the dataset is not externally accessible for inspection in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MS4UI, a dataset of 2,413 UI instructional videos from Adobe Creative Cloud products (167 hours) with three annotation layers: step-level video segmentation, per-step text instructions, and per-step key frame selection. The authors use GPT-3.5 on video transcripts to generate initial segmentations and summaries, which are then revised by human workers. Experiments with several text-only, vision-only, and multimodal baselines on the three tasks report performance close to random (e.g., BART-XSUM ROUGE-1=3.23 vs Random 2.77; MLASK Recall@1=10.46 vs Random 3.09), leading to the claim that existing methods struggle on UI instructional video summarization and that new methods are needed.","tokens_in":7817,"tokens_out":6548,"duration_ms":67638,"significance":"If the dataset and its annotations are of high quality, this is a useful contribution: it is likely the first benchmark for UI instructional videos that combines step segmentation, text summarization, and key frame selection, and it targets executable, fine-grained instructions rather than generic semantic summaries. The paper provides concrete tasks, metrics, and baseline results, and the near-random baseline performance indicates that the tasks are not trivially solved by current models. The main value is in the resource itself, so demonstrating annotation quality and reproducibility is essential. The paper does not yet ship machine-checked proofs or code; the dataset is promised in supplementary material but not citable or verifiable from the text.","major_comments":[{"comment":"Annotation quality is unmeasured: the paper reports no inter-annotator agreement, no annotator count, and no comparison of the GPT-3.5-plus-human labels against independent manual annotations. Since the central conclusion that \"existing methods struggle\" is inferred from low baseline scores relative to these labels, the conclusion is only meaningful if the reference labels are accurate. Please add quality statistics (e.g., boundary IOU agreement between annotators, ROUGE agreement between annotators, proportion of GPT-3.5 outputs changed by humans) and describe annotator qualifications, instructions, and the actual number of videos double-annotated. Also reconcile \"manually annotated\" in the abstract with the GPT-3.5-first pipeline described here.","section":"§3.2"},{"comment":"Results are reported without variance, confidence intervals, or significance tests. For claims that baselines are near random, it is important to know whether the difference between a baseline and the random baseline is within run-to-run variation. Please report means and standard deviations over multiple runs (at least three) and, where possible, paired significance tests (e.g., bootstrap or matched-pair).","section":"Tables 2–4"},{"comment":"The experimental protocol for baselines is underspecified. It is unclear which models are trained from scratch, fine-tuned, or used off-the-shelf; no hyperparameters, training steps, learning rates, or number of runs are given. In particular, for text summarization, BERT2BERT, T5, and BART variants are described as \"pre-trained language models\" but the paper later mentions a train/validation/test split, so it is ambiguous whether these models were fine-tuned on MS4UI. Please specify the exact adaptation procedure for each baseline so that the \"existing methods struggle\" claim can be reproduced and fairly attributed.","section":"§4.1 and §A.2"},{"comment":"The video summarization metric is not fully defined. Explain how candidate frames are sampled, how \"the image closest to the human-labeled key image frame\" is determined (by timestamp? by visual similarity?), how Recall@k aggregates over the multiple key frames per video, and what the Random baseline does in this setting. As written, Table 4's numbers cannot be interpreted by a reader who wants to implement the metric.","section":"§4.2"}],"minor_comments":[{"comment":"The abstract and introduction state the videos are \"manually annotated,\" but §3.2 reveals a GPT-3.5-first pipeline with human verification; rephrase to accurately describe the hybrid process.","section":"Abstract and §1"},{"comment":"The row label \"ASumm\" should be \"A2Summ\" for consistency with §4.1.","section":"Table 4"},{"comment":"The text says \"ROUGE-1, ROUGE-2 and ROUGE-2 metrics\"; the third metric should be ROUGE-L.","section":"§4.2"},{"comment":"The sentence \"existing video summarization methods show unsatisfoctory performance\" contains a typo: \"unsatisfoctory\" should be \"unsatisfactory.\"","section":"§7"},{"comment":"Step 4's timestamp \"1:72-1:22\" contains an invalid minute value; check the correct time range.","section":"Figure 1"},{"comment":"The Limitations section does not mention the potential impact of GPT-3.5-generated seed annotations on label quality or the lack of external validation; add a sentence acknowledging this risk and any plans for future reliability studies.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is a new benchmark, but the benchmark's credibility hinges on annotation quality, which is not demonstrated. I encourage the editor to request that the dataset be made publicly available with a persistent DOI and that the authors include annotation quality metrics and a clear provenance statement (GPT-3.5-assisted, not purely manual). The discrepancy between \"manually annotated\" and the GPT-3.5 pipeline may raise concerns about overclaiming and should be corrected. The paper has merit but needs substantial revision to substantiate the benchmark's validity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: this is a useful dataset paper, not a major breakthrough, and the main risk is that the annotations are less solid than the abstract implies. The novelty is real—combining step segmentation, per-step text instructions, and key-frame selection for UI instructional videos is new relative to How2 and TL;DW—and at 2,413 videos and 167 hours it is large enough to be a testbed. The examples in Figures 3 and 4 look genuinely executable, with precise UI terms and time-stamped steps. The baseline experiments are honest: every model is near random on ROUGE and Recall@1, which supports the claim that standard methods are not suited to this task.\n\nThe soft spots are real and worth fixing before the benchmark is trusted. The annotations are generated by GPT-3.5 and then verified by human workers, but the paper reports no annotator count, no inter-annotator agreement, and no quality-control metrics. The abstract calls the data 'manually annotated', which is misleading given the pipeline. This matters because the headline conclusion—existing methods struggle—could also be explained by noisy or systematically biased references (e.g., summaries that follow the transcript too literally). Relatedly, there are no error bars or significance tests on any table, and the tables have typos: 'ASumm' for 'A2Summ', 'T-5' for 'T5', and a duplicated 'segmentation' in the text summarization discussion. These are minor individually but suggest the paper was not proofread.\n\nThe 'first dataset' claim is probably defensible: TL;DW does instructional video summarization but not UI-specific, and How2 lacks key frames. Still, the wording 'first dataset containing segmentation and multi-modal summarization tasks' should be qualified to 'for UI instructional videos,' which it mostly is.\n\nWho is this for? Researchers in video summarization, UI understanding, and instruction following. The dataset, once released, could become a standard eval for UI-aware summarization. The annotation provenance needs measurement first. I would send this to serious peer review, with a request for annotation agreement metrics and dataset release, not desk-reject it.","headline":"Useful new dataset for UI video summarization, but the annotation provenance must be measured before the difficulty claims can be fully trusted.","tokens_in":8329,"tokens_out":2657,"would_cite":false,"duration_ms":30366,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces MS4UI, a dataset of 2,413 UI instructional videos, and shows that current multi-modal summarization methods perform poorly at producing step-by-step executable instructions.","keywords":["MS4UI dataset","UI instructional video summarization","multi-modal summarization","video segmentation","key frame selection","step-by-step instructions","benchmark"],"falsifier":"Have independent human annotators re-segment and re-summarize a random sample of MS4UI videos (say 50) and compare against the published ground truth. If boundary agreement falls below typical human thresholds (e.g., MIOU below 0.5) or summary ROUGE agreement is near the baselines' scores, then the conclusion that 'methods struggle' is confounded by label noise rather than task difficulty.","tokens_in":7412,"feed_emoji":"🖥️","tokens_out":6657,"duration_ms":65323,"temperature":0.7,"pith_summary":"This paper introduces MS4UI, a dataset of 2,413 user-interface (UI) instructional videos totaling 167 hours, with human-verified step segmentations, step-level text instructions, and key frames. It argues that existing multi-modal summarization benchmarks are designed for generic semantic summaries and cannot evaluate the step-by-step, executable instructions that UI tutorials require. The paper defines three core tasks—video segmentation, text summarization, and video summarization—with evaluation metrics, and reports that current state-of-the-art methods including BART, T5, MLASK, and A2Summ perform poorly on all three. If this is right, the field gains a benchmark that tests fine-grained, structured vision-and-language grounding in addition to summarization.","feed_headline":"Step-by-step UI video summarization stumps current models","feed_subtitle":"A new 2,413-video benchmark demands executable instructions plus key frames; today's best methods score near random.","key_machinery":"The central object is the MS4UI dataset itself, built by using GPT-3.5 to segment transcriptions into timestamped steps and draft step summaries, then having human annotators verify and correct boundaries and text, and choose one representative key frame per step. The task formulation chains segmentation, step-level summarization, and key-frame selection; the evaluation uses MIOU and thresholded F1 for segmentation, ROUGE-1/2/L for text, and Recall@1/2/5 for key-frame selection. The dataset is what carries the argument: because it provides aligned ground truth across all three modalities, it can quantify how far current methods are from usable, executable UI instructions.","core_discovery":"On its own terms, the paper's central claim is that MS4UI is the first dataset to combine step segmentation with text and video summarization for UI instructional videos, and that this combination exposes a clear gap in existing methods. The paper reports that the best text-based segmentation baseline reaches only a 20.53 MIOU, the best multi-modal summarizer (MLASK) reaches only 5.10 ROUGE-1, and the best key-frame selection (MLASK) reaches only 10.46 Recall@1. The authors interpret these results as evidence that UI videos require specific modeling of structured screen content and action-level grounding rather than generic semantic summarization.","pith_inferences":["I infer that the same three-task structure could transfer to other procedural domains (e.g., cooking, software onboarding, device setup), where step-level grounding matters as much as it does here.","The reliance on GPT-3.5 plus human verification suggests a scalable annotation recipe for video-step benchmarks; if future datasets adopt it, the key question will be how much quality control (e.g., agreement metrics) is needed to keep labels trustworthy.","The paper's finding that vision-based segmentation fails suggests that screen-text and layout features (like OCR or icon detection) may be more informative than generic appearance features for UI boundaries; this is a testable hypothesis the paper does not pursue.","A natural extension would be to test whether large language models with video or frame input can directly produce step summaries with timestamps, bypassing separate segmentation; the dataset's public split would let that be measured."],"forward_implications":["Existing multi-modal summarization systems will need UI-specific visual encoders that can read icons, panels, and fine-grained screen changes, not just object-level semantics.","The dataset provides aligned step boundaries, text, and frames, enabling future work on joint segmentation-and-summarization models rather than treating them as separate tasks.","The low baseline scores set a concrete starting point: any new method that beats MLASK on ROUGE and Recall@1 while keeping MIOU high would establish a measurable advance.","Instructional-video summarization can be evaluated for executability, not just semantic overlap, which shifts the goal from 'what the video is about' to 'what the user should do next.'","The 8:1:1 train/validation/test split supports reproducibility and gives the community a fixed setup for comparing future methods."],"supporting_citations":[{"why":"Represents general video summarization datasets (TVSum) that lack segmentation and text summaries, used to show the gap in task design.","marker":"Song et al., 2015"},{"why":"Represents another general video summarization dataset (SumMe), contrasted with MS4UI's step-level structure.","marker":"Gygli et al., 2014"},{"why":"News video multimodal summarization dataset (VMSMO) with text but no step segmentation, highlighting what MS4UI adds.","marker":"Li et al., 2020a"},{"why":"How2 dataset, the closest prior instructional-video benchmark with segmentation and text but no key frames.","marker":"Sanabria et al., 2018"},{"why":"Cross TextSeg, the text-based segmentation baseline that outperforms vision-based methods in the experiments.","marker":"Lukasik et al., 2020"},{"why":"LGSS, the vision-based scene segmentation baseline and the source of the MIOU evaluation protocol.","marker":"Rao et al., 2020"},{"why":"MLASK, the strongest multi-modal summarization baseline on text and video tasks, central to the 'methods struggle' claim.","marker":"Krubiński and Pecina, 2023"},{"why":"A2Summ, a multi-modal summarization baseline used to show that UI videos require specialized modeling.","marker":"He et al., 2023"}],"fun_headline_variants":["New dataset shows AI lags on UI video summarization","2,413 UI videos expose summarization model limits","Benchmark: UI video tutorials stump state-of-the-art models","MS4UI benchmark tests step-by-step UI video summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that the GPT-3.5-generated step segmentations and summaries, after human verification, are accurate enough to serve as ground truth; the paper reports no inter-annotator agreement, annotator count, or quality-control measurements.","fun_headline_variants_meta":{"raw":{"variants":["New dataset shows AI lags on UI video summarization","2,413 UI videos expose summarization model limits","Benchmark: UI video tutorials stump state-of-the-art models","MS4UI benchmark tests step-by-step UI video summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1327,"prompt_tokens":838,"completion_tokens":489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":422}},"tokens_in":454,"tokens_out":489,"duration_ms":5805,"temperature":1.0,"reasoning_tokens":422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:44:12.410734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human annotators re-segment and re-summarize a random sample of MS4UI videos (say 50) and compare against the published ground truth. If boundary agreement falls below typical human thresholds (e.g., MIOU below 0.5) or summary ROUGE agreement is near the baselines' scores, then the conclusion that 'methods struggle' is confounded by label noise rather than task difficulty.","supporting_citations":[],"review_version":1}