{"id":"ec49a02c-9cd5-4d37-9925-14aeaf4f241b","arxiv_id":"2504.18689","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"HierSum uses hierarchical attention over local subtitles and global instructions, trained with most-replayed statistics, to select important frames for video summarization.","lead":"This paper presents HierSum, a method for summarizing instructional videos by combining subtitle-level text, global written instructions, and YouTube's most-replayed viewer statistics to select important segments. It reports gains on some benchmark metrics, while the paper's own tables show it trailing prior methods on several headline F1 scores.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's claim of consistent state-of-the-art results is contradicted by the paper's own Tables 2 and 5: on TVSum F1 and on WikiHow F1/rank, HierSum is not best. This internal inconsistency is the load-bearing weakness of the central claim.","rationale":"The reader's weakest_assumption focuses on the validity of most-replayed statistics as a proxy for human importance. That is a real scientific concern, but it is partly a matter of external validity and could be addressed by a human-annotation study. The more immediately load-bearing problem is internal: the headline claim of consistent outperformance is contradicted by the paper's own Tables 2 and 5. A claim that is internally contradicted cannot be accepted even provisionally. The authors do include textual caveats in Section 4.5, acknowledging that on TVSum they 'slightly under-perform on F1' and that WikiHow baselines use a different protocol, but these caveats are not reflected in the abstract or conclusion, which both assert consistent SOTA performance. I therefore agree with the REJECT direction, though my route differs from the reader's weakest_assumption. If the authors rerun WikiHow with a matched protocol and still show gains, the claim could be narrowed to 'rank correlation on TVSum and F1/MAP on Mr.HiSum' and the paper might become conditionally acceptable after adding code/data and human validation of the replay signal.","tokens_in":14009,"tokens_out":3663,"duration_ms":33389,"concrete_test":"Add an experiment on the WikiHow test set in which HierSum scores are converted to shot-level scores using the same 32-frame/8-FPS segments and 0/1 knapsack protocol used by IV-Sum [38], then compute F1, Kendall's tau, and Spearman's rho under that protocol. If HierSum's F1 remains below IV-Sum's reported 67.3, the 'consistently outperforms' claim is refuted; if it rises above, the original Table 5 comparison was a protocol artifact and the claim could be repaired with explicit caveats.","verdict_should_be":"REJECT","load_bearing_attack":"Section 4.5 contains the paper's central evidence. Table 2 (TVSum) reports HierSum F1 = 62.5, below CLIP-It (64.2) and A2Summ (63.4); only Kendall's tau (0.172) and Spearman's rho (0.225) are best among the listed methods. Table 5 (WikiHow test set) reports HierSum trained on WikiHow with F1 = 48.17 and rank correlations tau = 0.08, rho = 0.10, far below IV-Sum (F1 = 67.3, tau = 0.101, rho = 0.212) and CLIP-It with ASR (F1 = 62.5, tau = 0.093, rho = 0.191). The authors note that the grayed baselines use a different protocol (32-frame segments at 8 FPS converted via 0/1 knapsack) and their own model uses top-55% frame selection, so the comparison may not be apples-to-apples. But the abstract nonetheless states without qualification that HierSum 'consistently outperforms existing methods in key metrics such as F1-score and rank correlation.' The central claim as written is therefore not supported by the paper's own numbers, independent of any external evaluation concerns.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HierSum, a hierarchical multimodal attention model for instructional video summarization. The model combines frame-level video features with subtitle-level local text and video-level global instruction text, and it is trained with an alternating parent-child protocol in which several local subtitle batches are followed by one global description batch. Supervision comes from YouTube \"most replayed\" statistics converted to binary importance labels and regression targets, combined with focal loss, an MSE loss on replay scores, and inter- and intra-modality contrastive losses. The authors evaluate on TVSum, BLiSS, Mr.HiSum, and WikiHow, and they additionally curate a new multi-modal instructional dataset from WikiHow and EHow videos. The central claim, stated in the abstract and conclusion, is that HierSum consistently outperforms existing methods on F1-score and rank correlation, with roughly 1-2% gains from pre-training on the curated dataset.","tokens_in":14239,"tokens_out":5477,"duration_ms":51848,"significance":"If the central claims were fully supported, the paper would make a useful contribution by demonstrating that hierarchical alignment of fine-grained subtitle cues with global instructional text, combined with replay-statistic supervision, improves video summarization. The proposed parent-child training protocol is a reasonable and timely idea, and the cross-dataset ablations in Tables 6 and 7, together with the global-step frequency study in Figure 3, provide useful design insights. The authors also report qualitative failure cases, which is a strength. However, the evidence as presented supports only a narrow set of claims: rank-correlation gains on TVSum, F1/MAP gains on Mr.HiSum, and cosine-similarity gains on BLiSS. The broad \"consistently outperforms\" statement is not supported by the paper's own tables.","major_comments":[{"comment":"The abstract and conclusion claim that HierSum \"consistently outperforms existing methods in key metrics such as F1-score and rank correlation,\" but the paper's own numbers contradict this. On TVSum (Table 2), HierSum's F1 of 62.5 is below CLIP-It (64.2), iPTNet (63.4), and A2Summ (63.4); only the rank correlations are best among the comparable methods. On WikiHow (Table 5), HierSum trained on WikiHow achieves F1 48.17 and rank correlations 0.08/0.10, far below IV-Sum (67.3, 0.101, 0.212) and CLIP-It with ASR (62.5, 0.093, 0.191). The abstract and conclusion must be revised to specify exactly which datasets and metrics show improvements, or additional experiments must be provided to support the unqualified claim.","section":"Abstract and Section 7 (Conclusion) vs. Tables 2 and 5"},{"comment":"The WikiHow comparison is not apples-to-apples. The text states that the grayed-out baselines use segments of 32 frames sampled at 8 FPS and convert frame-level scores to shot-level scores via 0/1 knapsack, while HierSum selects the top 55% of frames. Under these different protocols, the IV-Sum and CLIP-It numbers in the same table cannot be directly compared with HierSum's numbers. To support any claim of superiority on WikiHow, the authors must either rerun the baselines under the same segment-and-selection protocol or explicitly present the comparison as protocol-different and refrain from drawing comparative conclusions from it. As written, the only valid same-protocol comparison is HierSum trained on pseudo summaries versus HierSum trained on the curated WikiHow data, which supports the pre-training contribution but not state-of-the-art performance.","section":"Section 4.5, Table 5 (WikiHow)"},{"comment":"All reported results are single numbers without variance, confidence intervals, or significance tests. The claimed improvements are small, e.g., Kendall's tau 0.172 versus A2Summ's 0.150 on TVSum and F1 58.16 versus PGL-SUM's 56.89 on Mr.HiSum, and these differences could easily be within run-to-run noise. The paper should report results over multiple random seeds or provide bootstrap confidence intervals for the main metrics, and the word \"significantly\" in the abstract and contribution list should be avoided unless a statistical test supports it.","section":"Section 4.5, Tables 2-5"},{"comment":"The Mr.HiSum evaluation is partly circular. The Mr.HiSum ground-truth importance scores are themselves constructed from YouTube most-replayed statistics, which is the same signal used as HierSum's supervision. Table 4 therefore largely measures how well the model fits its training signal rather than how well the summaries agree with independent human judgments. The zero-shot transfer results in Table 7 (F1 44.7 versus 57.7 after fine-tuning) also show that the learned scores do not transfer cleanly to Mr.HiSum's Inception-feature domain. The paper should explicitly acknowledge this circularity and provide a validation that does not rely on the same statistic, such as a small human study or an evaluation on a human-annotated benchmark like SumMe.","section":"Section 3 (Most Replayed Statistics) and Section 4.1 (Mr.HiSum), Tables 4 and 7"}],"minor_comments":[{"comment":"The text contains the typo \"HierSumm\" in the TVSum paragraph; it should read \"HierSum.\"","section":"Section 4.5"},{"comment":"The DSNet-AF row reports rho as \".0138\", which appears to be a typo for \"0.138\" or similar; please correct.","section":"Table 2"},{"comment":"The text says hyperparameters α, β, and γ control the loss trade-off, but Equation (4) also includes λ for the intra-modality contrastive loss; the list of hyperparameters should be completed and their values given consistently with Table 1, which currently reports only β and λ.","section":"Equation (4) and surrounding text"},{"comment":"The dataset curation description uses terms such as \"scrape\" and \"crawl\" but does not report the final number of videos, the filtering statistics, or a train/validation/test split; please add these details for reproducibility.","section":"Section 4.2"},{"comment":"The dataset name is spelled inconsistently as both \"BLISS\" and \"BLiSS\"; please standardize to \"BLiSS\" as used in the dataset references and tables.","section":"Throughout"},{"comment":"The caption mentions \"ASR only\" but the figure legend and axis labels are not fully defined; please add a clear legend and describe the meaning of step = 0 and step = 1 in the caption.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a borderline case: the architectural idea is reasonable and the ablation study is informative, but the main claims are overstated relative to the reported results. I recommend major revision rather than rejection because the claims can be corrected and the evaluation gaps are addressable. The editor may wish to ask the authors to focus their revision on the WikiHow evaluation protocol and on adding variance or significance information for the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the method is genuinely well put together: a hierarchical parent-child training scheme that blends A2Summ-style alignment-guided self-attention and dual contrastive losses with Mr.HiSum's most-replayed supervision, plus a newly curated WikiHow/EHow video dataset. The writing is clear, the ablations are useful (cross-dataset transfer, global-step interval), and the authors even retrained A2Summ with official code to keep the BLiSS comparison fair. That is real care. Second, the abstract says HierSum \"consistently outperforms existing methods in key metrics such as F1-score and rank correlation,\" and that is simply not what the paper's own tables show. On TVSum, HierSum ranks third on F1 (62.5 vs CLIP-It's 64.2 and A2Summ's 63.4); on WikiHow, it is far behind IV-Sum (F1 48.17 vs 67.3) and CLIP-It with ASR (62.5). The body is actually honest about this: Table 2 notes the F1 underperformance, and Table 5 carefully flags the different evaluation protocols (32-frame segments vs top-55% frame selection). But the abstract and conclusion do not carry that nuance, and the central claim as stated is contradicted by the evidence in the same paper.\n\nThe soft spots are mostly fixable rather than fatal. There are no error bars, variance, or significance tests anywhere; every table is a single number. The most-replayed supervision is posited as correlating with summary-worthy content, but that correlation is never validated against human importance judgments, and on Mr.HiSum the ground truth is itself built from the same statistic, so those results are partly circular. The WikiHow comparison is genuinely apples-to-oranges, and the authors acknowledge it, but then they still present the numbers without the caveat in the abstract. No code or dataset is released yet, so independent verification is impossible.\n\nWho gets value? A knowledgeable reader working on multimodal summarization will find the hierarchical combination and the dataset idea worth thinking about, and the ablation on global-step frequency is a useful datapoint. But the paper in its current form needs a rewrite of the claims, a proper uncertainty analysis, and ideally at least a partial release of the dataset before it can stand as published work.\n\nMy recommendation: do not desk-reject, but send it to serious peer review. The method is substantive and the authors show scholarly honesty in the body, so a good referee could push them to align the claims with the evidence and add the missing baselines and significance testing. The paper is not there yet, but it is not a throwaway either.","headline":"A clearly described, honestly written method paper whose abstract overclaims: the own tables show HierSum is not consistently SOTA, so the central claim needs rewriting before it can be published.","tokens_in":736,"tokens_out":968,"would_cite":false,"duration_ms":25647,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hierarchical attention over subtitles and global instructions, supervised by YouTube replay statistics, improves instructional video summarization.","keywords":["video summarization","instructional videos","hierarchical attention","most replayed statistics","multi-modal learning","parent-child training","subtitle alignment","global instructions"],"falsifier":"Take the Mr.HiSum or curated dataset's most-replayed labels and compare them, shot by shot, with human importance ratings collected for the same videos; if agreement is low, or if a model trained on human ratings outperforms HierSum on human-judged summarization metrics while HierSum only wins on replay-derived ground truth, the central claim is refuted.","tokens_in":13715,"feed_emoji":"🎬","tokens_out":5202,"duration_ms":48845,"temperature":0.7,"pith_summary":"The paper tries to establish that video summarization for instructional videos improves when a model learns from two levels of text at once: fine-grained subtitle sentences aligned to moments of the video, and coarse global instructions describing the whole task. The proposed HierSum model alternates between these two levels in a parent-child training loop and uses YouTube's \"most replayed\" statistics as the supervisory signal that tells it which segments matter. If this works, summaries of how-to videos would better preserve the actual steps of a task rather than just the visually salient or frequently repeated moments. The paper also contributes a new multi-modal dataset of WikiHow and EHow videos paired with step-by-step articles. Reported gains include roughly 2 percent on TVSum rank correlation, 2 percent F1 on Mr.HiSum, and 1 percent cosine similarity on BLiSS, though F1 on TVSum trails the best baseline.","feed_headline":"How-to video summaries improve with two-level text attention","feed_subtitle":"HierSum trains on subtitles plus global instructions, using replay stats as labels, and gains on rank correlation and F1.","key_machinery":"The machinery is a parent-child hierarchical training protocol over a shared multimodal transformer. In child steps the input sequence interleaves video frame features with sentence embeddings of subtitles; in parent steps the subtitles are replaced by one global description and only video importance is predicted. An alignment-guided self-attention mask forces cross-modal attention between frames and their time-aligned subtitles, and the final objective combines focal loss on importance labels, mean-squared error on most-replayed scores, and inter- and intra-modality contrastive losses. This two-level loop is what carries the claim: it is the mechanism by which local step cues and global task context jointly shape the importance scores.","core_discovery":"HierSum's central claim is that a shared transformer trained with an alternating protocol, using m batches of clip-level subtitle-video pairs followed by one batch of global instruction-video pairs, learns representations that capture both the local, step-by-step execution of a task and its overall goal. The model predicts frame importance and sentence importance, with an alignment-guided self-attention mask that lets video frames attend to their corresponding subtitles, and it is trained with a loss combining focal classification, mean-squared error on most-replayed scores, and inter- and intra-modality contrastive terms. Using most-replayed statistics as the relevance label, the paper curates a large instructional-video dataset and shows that pre-training on it transfers to TVSum, BLiSS, Mr.HiSum, and the WikiHow test set. The intended consequence is that summaries align better with human judgment, as reflected in rank correlation gains, while remaining competitive on F1.","pith_inferences":["If most-replayed statistics are a valid proxy for instructional importance, the same two-level text supervision could apply to other procedural media, such as repair or medical videos, where step descriptions and global protocols coexist.","The released dataset, pairing videos with both ASR subtitles and step-by-step articles, could be reused for step grounding or temporal action segmentation, not just summarization.","The optimal ratio of local to global training steps is likely dataset-dependent, so future work might adaptively schedule global steps instead of fixing them."],"forward_implications":["Training on the curated WikiHow-EHow dataset, using most-replayed labels plus both text levels, transfers to existing benchmarks and yields roughly 1 to 2 percent gains after fine-tuning.","The parent-child protocol is crucial: using only subtitles or only global instructions hurts performance, and the best results appear around one global batch per five local batches.","Most-replayed supervision transfers to summarization: a model trained to predict replay scores also produces competitive summaries on F1 and mean average precision.","The model improves rank-order agreement with human importance scores on TVSum, raising Kendall's tau and Spearman's rho over the compared baselines."],"supporting_citations":[{"why":"Supplies the alignment-guided self-attention mask, the CLS-token sequence construction, and the intra-modality contrastive loss that HierSum adapts.","marker":"[21]"},{"why":"Provides the most-replayed supervision paradigm and the Mr.HiSum dataset used for training and evaluation.","marker":"[49]"},{"why":"Provides the WikiHow test set, pseudo-summary baselines, and the prior instructional-summarization approach HierSum extends.","marker":"[38]"},{"why":"Provides the TVSum benchmark and the F1 and rank-correlation evaluation protocol.","marker":"[48]"},{"why":"Supplies the CLIP video and text encoders used to featurize frames and instructions.","marker":"[44]"},{"why":"Supplies the Sentence-BERT text encoder used for subtitle and description representations.","marker":"[46]"}],"fun_headline_variants":["HierSum: Two-level attention lifts video summarization","Local subtitles, global instructions: HierSum sums it up","Replay stats guide HierSum to better video summaries","Alternating local and global text improves video summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that YouTube's \"most replayed\" statistic marks the same segments a good summary should keep; if viewers rewatch entertaining moments rather than instructive steps, the training labels drift away from what a summary needs, and the paper does not validate this correlation against human importance scores.","fun_headline_variants_meta":{"raw":{"variants":["HierSum: Two-level attention lifts video summarization","Local subtitles, global instructions: HierSum sums it up","Replay stats guide HierSum to better video summaries","Alternating local and global text improves video summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":2929,"prompt_tokens":913,"completion_tokens":2016,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":1952}},"tokens_in":529,"tokens_out":2016,"duration_ms":14273,"temperature":1.0,"reasoning_tokens":1952,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:12:29.934807+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the Mr.HiSum or curated dataset's most-replayed labels and compare them, shot by shot, with human importance ratings collected for the same videos; if agreement is low, or if a model trained on human ratings outperforms HierSum on human-judged summarization metrics while HierSum only wins on replay-derived ground truth, the central claim is refuted.","supporting_citations":[{"cited_title":"Align and attend: Multimodal summarization with dual contrastive losses","cited_arxiv_id":null,"evidence_quote":"Supplies the alignment-guided self-attention mask, the CLS-token sequence construction, and the intra-modality contrastive loss that HierSum adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the most-replayed supervision paradigm and the Mr.HiSum dataset used for training and evaluation."},{"cited_title":"Tl; dw? summarizing instructional videos with task relevance and cross-modal saliency","cited_arxiv_id":null,"evidence_quote":"Provides the WikiHow test set, pseudo-summary baselines, and the prior instructional-summarization approach HierSum extends."},{"cited_title":"Tvsum: Summarizing web videos using titles","cited_arxiv_id":null,"evidence_quote":"Provides the TVSum benchmark and the F1 and rank-correlation evaluation protocol."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP video and text encoders used to featurize frames and instructions."},{"cited_title":"Sentence-bert: Sentence embeddings using siamese bert-networks","cited_arxiv_id":null,"evidence_quote":"Supplies the Sentence-BERT text encoder used for subtitle and description representations."}],"review_version":1}