{"id":"623d2db0-cf2f-43e0-abd8-872e9025ee84","arxiv_id":"2506.23714","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A multimodal extractive summarizer using text, prosody, and facial cues reports higher ROUGE, BLEU, and video-selection F1 than an Edmundson baseline on short ChaLearn interview clips.","lead":"This paper combines transcripts, voice prosody, and facial cues to select the most important sentences in short interview videos, then highlights words emphasized in several modalities. It reports large gains over a text-only baseline, but the comparison is confounded by summary length and uses a machine-generated reference.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed gains are confounded by unequal summary lengths: proposed summaries are ~83% longer than Edmundson's (length ratio 0.5734 vs 0.3125), so higher ROUGE/BLEU/F1 may reflect length, not multimodal quality.","rationale":"The reader's REJECT verdict is sound, but the single most load-bearing flaw is not primarily the validity of the pseudo-ground truth; it is the uncontrolled summary length difference between the two compared methods. Even if GPT-4.5 pGT summaries were perfect human-quality references, the reported improvements would still be uninterpretable because the proposed method produces summaries nearly twice as long. ROUGE, BLEU, and segment-overlap F1 all increase trivially with the amount of content included. The paper itself reports length ratio in Table 1, so the data needed to spot this confound are present. The reader's concern about pGT validity is also serious: the paper's own limitation statement acknowledges LLM bias, and the internal consistency check (Jaccard 0.78) plus a 50-clip qualitative review does not substitute for human annotation of the full test set. However, the length mismatch is more immediately fatal because it invalidates the comparative claim under the paper's own evaluation protocol. The ablation study (Table 3) does not resolve this, since all variants are evaluated at whatever length the threshold produces. No code or data are provided, so the reported numbers cannot be independently reproduced. A length-matched re-evaluation is the minimal check that would settle whether the multimodal framework adds anything beyond selecting more content. If the advantage persists at matched length, this particular objection is answered; if not, the headline claim collapses. The verdict should remain REJECT, not because the authors are careless, but because the evidence as presented does not support the claimed significant improvement.","tokens_in":10774,"tokens_out":3273,"duration_ms":38478,"concrete_test":"Recompute Tables 1 and 2 after constraining both methods to a fixed summary budget: for each clip, force Edmundson and the proposed method to select exactly the same number of sentences or words (e.g., the proposed method's 0.5734 length ratio), then recompute ROUGE-1/L, BLEU, BERTScore, F1, Kendall's tau, and Spearman's rho. Also add a length-matched random-selection baseline and a first-k-sentences baseline. If the proposed method no longer beats Edmundson at matched length, the reported gains are artifacts of unequal compression.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Tables 1 and 2 compare summaries of very different lengths. In Table 1, the proposed method has length ratio 0.5734, while Edmundson has 0.3125, i.e., the proposed summaries are roughly 83% longer. ROUGE-1/2/L, BLEU, and BERTScore are computed against the same pGT reference, and all are sensitive to candidate length: longer candidates have more opportunity to overlap reference content. The reported ROUGE-1 jump from 0.4769 to 0.7929 and BLEU jump from 0.2259 to 0.6411 may therefore reflect compression ratio rather than the contribution of multimodal cues. The video metrics are similarly length-sensitive: with matches defined by IoU > 0.5, selecting more frames or longer segments inflates recall and can inflate F1 (0.5709 to 0.6995). No length-matched baseline, precision-recall curve, or density/compression-adjusted metric is reported, and Algorithm 1's threshold (theta = mu + 0.3 sigma) does not control output length. Consequently, the central comparative claim is not established even if the pGT references are accepted as valid.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a multimodal, extractive video summarization pipeline for short interview videos. It combines textual cues (TF-IDF, sentiment, named entities), audio prosodic cues (pitch, loudness, voice quality), and visual cues (head movement, facial emotion transitions) into 'bonus words' that are weighted when scoring sentences. An adaptive threshold (θ = μ + λσ) and a cosine-similarity diversity penalty select sentences, which are then mapped to timestamped video segments. The system is evaluated on 1,500 clips from the ChaLearn First Impressions dataset against pseudo-ground-truth (pGT) summaries generated by GPT-4.5 with an extractive prompt. The authors report large gains over the Edmundson baseline in ROUGE, BLEU, BERTScore, and video-selection F1, and they include an ablation study. The paper acknowledges limitations of pGT evaluation and reports a small qualitative validation and a consistency check.","tokens_in":10980,"tokens_out":5678,"duration_ms":58048,"significance":"If the central claim were established, the paper would offer a cheap, modular way to inject behavioral cues into extractive summarization, and the notion of 'bonus words' is an interesting operationalization of cross-modal emphasis. The manuscript describes its pipeline clearly and reuses publicly available components (Whisper, MFA, MediaPipe, DeepFace, openSMILE), which supports reproducibility. It also explicitly acknowledges the limitations of pseudo-ground-truth evaluation and attempts a consistency check. However, the significance is currently limited because the evaluation does not isolate the effect of multimodal cues from summary length, and the reference standard is self-generated rather than independently human-validated.","major_comments":[{"comment":"The central comparative claim is confounded by summary length. The proposed method has length ratio 0.5734 versus 0.3125 for Edmundson, i.e., approximately 83% longer. ROUGE-1/2/L, BLEU, and BERTScore are all sensitive to candidate length: a longer candidate has more opportunities to match pGT tokens and phrases. No length-matched baseline, compression-adjusted metric, or precision-recall curve is reported. The video F1 comparison is similarly affected because matches defined by IoU > 0.5 reward longer selected intervals. Therefore the abstract and Section 4.2 claim that the framework 'significantly outperforms' Edmundson is not supported by the reported numbers.","section":"§4.2, Tables 1–2"},{"comment":"All text and video metrics are computed against pseudo-ground-truth summaries produced by GPT-4.5 with an extractive prompt. Because the proposed method is also a sentence-extractive selector, the evaluation essentially measures agreement with the LLM's selection rather than summarization quality. The two-part validation (50 manually inspected summaries; average Jaccard similarity of 0.78 between prompt variants) is helpful but does not replace human relevance judgments or an external benchmark. Without such a reference, the claimed improvement is not established; I am not claiming the bonus-word construction is circular, but the benchmark is self-generated and may be biased.","section":"§4.1"},{"comment":"The ablation results contradict the narrative. The multimodal system has BLEU 0.6411, while Visual-only achieves 0.7234 and w/o Audio achieves 0.7456; the statement that 'all single-modality variants clearly underperform compared to multimodal approaches' is false for BLEU. The absence of standard deviations and significance tests means that other differences (e.g., ROUGE-1 0.6743 vs. 0.7929) could also be within noise. Please report per-video or per-fold variance, significance tests, and length ratios for all variants.","section":"§4.3, Table 3"},{"comment":"Several free parameters are hand-set without validation: λ = 0.3 in Eq. (2), the diversity factor δ = 0.2, the head-movement displacement threshold, and the audio Z-score threshold. These directly control how many sentences are selected and thus the summary length. Without a validation split, sensitivity analysis, or parameter sweeps, the reported gains could be artifacts of threshold choices. In addition, λ is used both for the head-movement threshold and for the adaptive threshold factor, which is confusing.","section":"§3, Algorithm 1, Eq. (2)"}],"minor_comments":[{"comment":"The caption refers to 'ChaLearn'17', but the cited dataset is the ChaLearn LAP 2016 First Impressions challenge; please make the year consistent.","section":"Figure 1 caption"},{"comment":"The definition of bonus words is informal: it is unclear whether a sentence weight is the number of bonus-word occurrences or the number of distinct bonus words; Algorithm 1's Count(w, Words) suggests the former, which can double-count repeated terms.","section":"Section 3, 'Bonus words'"},{"comment":"The consistency check reports an average Jaccard similarity of 0.78 but does not state the number of videos sampled or the exact prompt variations used; please provide this detail.","section":"§4.1"},{"comment":"Table 3 does not report length ratios for the ablation variants, making it impossible to interpret BLEU/ROUGE differences across variants, especially in light of the length confounding raised in the major comments.","section":"Tables 1 and 3"},{"comment":"There is a typo 'psuedo-ground truth' in the video-based evaluation paragraph; it should read 'pseudo-ground truth'.","section":"§4.2"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical claim is not supported by the current evaluation because of the summary-length confound and the self-generated pseudo-ground-truth reference. Substantial new experiments (human annotations, length-controlled baselines, significance testing, and ideally evaluation on standard datasets such as SumMe or TVSum) would be needed before the paper could be considered for publication. The ablation table also contains an internal inconsistency that would need to be resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a workshop paper with a testable idea—bonus words from cross-modal emphasis—but the reported gains are not supported because the proposed summaries are roughly 83% longer than the Edmundson baseline and every metric is computed against a GPT-4.5 pseudo-ground truth. The reject verdict from the reader is right, and the length confound is the more damaging flaw.\n\nWhat is actually new: the bonus-word mechanism is a clean, small extension of Edmundson-style scoring: you detect word-level cues from prosody, head movement, and facial emotion transitions, then boost sentences containing words that are emphasized in at least one other modality. The paper is honest about the pGT limitation, does a 50-sample manual check and a Jaccard consistency test, and runs ablations. For a workshop, that is decent scientific hygiene.\n\nWhere it falls down: the comparison in Tables 1 and 2 is between summaries of different lengths. Length ratio 0.5734 vs 0.3125 means the proposed summaries have about 83% more chance to overlap the reference, which inflates ROUGE, BLEU, and IoU-based F1. No length-matched baseline, no precision-recall curves, no compression-adjusted metric. Also, all text and video metrics are against one self-generated LLM reference; even if the length issue were fixed, this would only show agreement with GPT-4.5's selection, not better summarization. There are no standard deviations or significance tests, no comparison against any recent multimodal summarizer, and no code or data released. The ablation table has an oddity: the full multimodal model's BLEU (0.6411) is lower than several of its own ablations (0.7456, 0.7234, 0.6895); the authors attribute this to BLEU's preference for exact n-gram matches, which is plausible but undercuts the 'consistent improvement' claim.\n\nWho gets value: readers working on cheap behavioral cues for summarization of short conversational video might find the bonus-word idea worth a look as related work. But the evaluation in this form does not establish the claim.\n\nRecommendation: reject in current form. If resubmitted with length-controlled baselines, a subset of human-annotated references, significance tests, and code, it could become a reasonable short paper. I would not spend a full review cycle on this version.","headline":"The bonus-word idea is worth a look, but the evaluation is confounded by summary length and a self-generated LLM reference, so the headline numbers don't carry.","tokens_in":11554,"tokens_out":4244,"would_cite":false,"duration_ms":46856,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that scoring transcript sentences with prosodic and facial emphasis cues, not just word frequency, lifts timestamped video summaries from 0.4769 to 0.7929 ROUGE-1 and improves segment-selection F1 by about 23 percent on…","keywords":["multimodal video summarization","computer vision","natural language processing","audio and speech processing","multimedia content creation","bonus words","prosodic features","pseudo-ground truth"],"falsifier":"Take a random sample of the same interview videos, have human annotators mark the most important sentences and segments, and recompute ROUGE, BERTScore, and segment F1 against those human references. If the multimodal method no longer beats Edmundson, or if its advantage shrinks to statistical noise, the paper's central claim is falsified. A direct check of the verbosity confound would be to equalize summary length between the two methods before comparing, since the proposed summaries are nearly twice as long (length ratio 0.5734 vs 0.3125), which can mechanically raise recall-style metrics such as ROUGE-1.","tokens_in":1684,"feed_emoji":"🎬","tokens_out":2684,"duration_ms":82618,"temperature":0.7,"pith_summary":"This paper tries to establish that a behavior-aware, multimodal video summarization pipeline—fusing text keywords with prosodic and facial-emphasis cues—produces better timestamped summaries than text-only extractive methods. The motivation is practical: interviews, lectures, and meetings carry meaning in tone and expression, not just words, and conventional summarizers ignore those signals. The authors evaluate on 1,500 short interview clips from the ChaLearn First Impressions dataset, using LLM-generated extractive pseudo-ground-truth summaries as references, and report large gains over the Edmundson baseline across text and video metrics. A sympathetic reader would take the paper's contribution to be the 'bonus word' mechanism and the demonstration that multimodal emphasis aligns with summarization importance.","feed_headline":"Adding audio and face cues lifts video summary scores by 30 points","feed_subtitle":"Scoring sentences with pitch, emotion, and head-movement emphasis beats text-only extraction on short interview videos.","key_machinery":"The load-bearing mechanism is the bonus-word scoring pipeline. A unified timeline, built with forced alignment, fuses textual cues (TF-IDF, sentiment analysis, named entities), prosodic cues (pitch, RMS loudness, Hammarberg index), and visual cues (nose-landmark displacement, facial emotion transitions); each transcript sentence is weighted by the frequency of bonus words, selected with an adaptive threshold $\\theta = \\mu + \\lambda\\sigma$, and diversified through a TF-IDF cosine-similarity penalty. The selected sentences are mapped to frame ranges and compiled into a subtitled summary video, while a separate GPT-4.5 extractive prompt generates pseudo-ground-truth summaries from the same timestamped transcripts for evaluation.","core_discovery":"The central claim is that a word's summarization importance is not purely lexical: when a transcript word coincides in time with a prosodic spike (pitch, loudness, tonality) or a visual cue (head movement, facial emotion transition), it becomes a 'bonus word' and should be weighted more heavily in sentence scoring. Combining this multimodal weighting with an adaptive threshold and a diversity penalty produces text summaries and timestamped video segments that, on the ChaLearn interview-clip subset, beat Edmundson's classical keyword-position-cue method on every reported metric: ROUGE-1 0.7929 vs 0.4769, BERTScore 0.9536 vs 0.9152, BLEU 0.6411 vs 0.2259, and video segment F1 0.6995 vs 0.5709.","pith_inferences":["Beyond the paper: the reported ROUGE gains could be partly driven by verbosity, since the proposed summaries are nearly twice as long as Edmundson's (length ratio 0.5734 vs 0.3125), so an exact-length-matched comparison would separate content selection from summary length.","Beyond the paper: the bonus-word mechanism should transfer to meetings, lectures, and vlogs where emphasis signals align with spoken keywords, but longer-form videos may stress the timestamp-alignment precision and threshold choices.","Beyond the paper: a natural next test is human evaluation of summary usefulness and fluency, because the pseudo-ground-truth references capture only extractive sentence selection, not whether viewers actually find the summaries more informative or expressive.","Beyond the paper: the fixed threshold factor $\\lambda=0.3$ and diversity factor $\\delta=0.2$ deserve sensitivity analysis, since the claimed improvements could depend on these specific parameter values."],"forward_implications":["If the framework generalizes to other conversational video, timestamped summaries will preserve emotional and prosodic emphasis rather than only lexical content.","The ablation indicates visual cues are the strongest single modality and removing them hurts performance most, so systems without face and pose analysis will underperform on expressive interview-style content.","The bonus-word mechanism offers a lightweight way to inject multimodal salience into classic extractive scoring without training a deep model.","The pseudo-ground-truth evaluation strategy provides a scalable path for assessing video summarization when human annotations are unavailable, subject to the paper's own caveats about validation."],"supporting_citations":[{"why":"Supplies the ChaLearn First Impressions dataset of short interview clips used for all experiments and evaluations.","marker":"[24]"},{"why":"Defines the Edmundson heuristic method that serves as the baseline and as the scoring skeleton the multimodal pipeline extends.","marker":"[7]"},{"why":"Provides the Whisper speech recognizer that produces the transcripts feeding the textual modality.","marker":"[25]"},{"why":"Supplies the Montreal Forced Aligner that gives word-level timestamps central to cross-modal fusion.","marker":"[27]"},{"why":"Motivates the LLM-based extractive pseudo-ground-truth summary generation used as the evaluation reference.","marker":"[35]"},{"why":"Also motivates the pseudo-ground-truth and cross-modal saliency strategy for video summarization.","marker":"[36]"},{"why":"Provides the MediaPipe pose model whose nose landmark is used for head-movement visual cues.","marker":"[29]"},{"why":"Provides the DeepFace facial emotion detector whose emotion transitions are used as visual cues.","marker":"[30]"},{"why":"Supplies the YAAPT pitch-tracking algorithm used for prosodic audio cues.","marker":"[31]"},{"why":"Supplies the openSMILE features for loudness and voice quality cues.","marker":"[32]"}],"fun_headline_variants":["Prosody and face cues sharpen video summaries","Multimodal bonus words boost video summary scores","Pitch and head movement cues improve summarization","Cue fusion lifts video summary F1 by 23%"],"cache_read_input_tokens":13696,"weakest_assumption_plain":"The load-bearing premise is that the GPT-generated pseudo-ground-truth summaries are a valid stand-in for human judgments of which sentences matter; if those references are biased, the reported improvements do not show better summarization.","fun_headline_variants_meta":{"raw":{"variants":["Prosody and face cues sharpen video summaries","Multimodal bonus words boost video summary scores","Pitch and head movement cues improve summarization","Cue fusion lifts video summary F1 by 23%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1441,"prompt_tokens":934,"completion_tokens":507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":550,"tokens_out":507,"duration_ms":5998,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:33:24.038215+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the same interview videos, have human annotators mark the most important sentences and segments, and recompute ROUGE, BERTScore, and segment F1 against those human references. If the multimodal method no longer beats Edmundson, or if its advantage shrinks to statistical noise, the paper's central claim is falsified. A direct check of the verbosity confound would be to equalize summary length between the two methods before comparing, since the proposed summaries are nearly twice as long (length ratio 0.5734 vs 0.3125), which can mechanically raise recall-style metrics such as ROUGE-1.","supporting_citations":[{"cited_title":"Ponce-López, B","cited_arxiv_id":null,"evidence_quote":"Supplies the ChaLearn First Impressions dataset of short interview clips used for all experiments and evaluations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Edmundson heuristic method that serves as the baseline and as the scoring skeleton the multimodal pipeline extends."},{"cited_title":"McAuliffe, M","cited_arxiv_id":null,"evidence_quote":"Supplies the Montreal Forced Aligner that gives word-level timestamps central to cross-modal fusion."},{"cited_title":"Scaling Up Video Summarization Pretraining with Large Language Models","cited_arxiv_id":"2404.03398","evidence_quote":"Motivates the LLM-based extractive pseudo-ground-truth summary generation used as the evaluation reference."},{"cited_title":"Narasimhan, A","cited_arxiv_id":null,"evidence_quote":"Also motivates the pseudo-ground-truth and cross-modal saliency strategy for video summarization."},{"cited_title":"Kasi, Yet another algorithm for pitch tracking (yaapt), 2002","cited_arxiv_id":null,"evidence_quote":"Supplies the YAAPT pitch-tracking algorithm used for prosodic audio cues."},{"cited_title":"Eyben, M","cited_arxiv_id":null,"evidence_quote":"Supplies the openSMILE features for loudness and voice quality cues."}],"review_version":1}