{"id":"e5db9a59-28e4-426f-8746-e80aed5bfcae","arxiv_id":"2412.01798","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SEAL compresses long videos into scene, object, and action tokens, then selects a query-relevant, diverse subset, reporting improvements on LVBench, MovieChat-1K, and Ego4D.","lead":"This paper compresses hour-long videos into a small set of scene, object, and action tokens, then selects the most relevant and diverse tokens for each question. The authors report top results on long-video QA and temporal grounding benchmarks, including LVBench and Ego4D.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LVBench headline may be answer-option leakage: q in Eq. 3.2 is never stated to exclude the four MCQ options, so the BLIP-2 relevance scorer can select tokens using the correct answer.","rationale":"The reader's weakest assumption identifies the same load-bearing concern I see, and it is the most direct threat to the paper's headline claim. The LVBench comparison is the strongest quantitative evidence for SEAL, and the paper does not document whether the answer options are removed from the query used by the attention-learning relevance scorer. If they are not removed, the method gains access to the correct answer during token selection, which is a form of label leakage even though the options are available to the final MLLM. This would inflate the reported accuracy and undermine the claim that SEAL's representation, rather than answer-guided retrieval, drives the improvement. Other issues, such as the undescribed subset-selection solver and missing error bars, are secondary because they affect reproducibility more than the validity of the central comparison. I do not recommend rejection because the concern is testable and the other two benchmarks (MovieChat-1K open-ended QA and Ego4D grounding) do not have this multiple-choice leakage mechanism, so the method may still have value. A conditional acceptance with a mandatory ablation separating question-only from question-plus-options queries is the appropriate outcome, matching the reader's verdict.","tokens_in":17326,"tokens_out":6068,"duration_ms":70408,"concrete_test":"Run LVBench with two selection conditions: (A) q = full prompt with four options; (B) q = question text only, with options still passed to the MLLM for the final answer. Use identical token budgets, alpha, and MLLM prompt. If overall accuracy drops by more than 2-3 points from (A) to (B), the headline LVBench gain is substantially driven by answer-option leakage; report per-task deltas for KIR, ER, and TG to locate the effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines the selection objective F_s with R(t_s, q), where q is the query. The paper never specifies whether q contains the full LVBench prompt (question plus four options) or only the question. The supplementary prompt (D.2) shows the MLLM receives question and options; if the same q is fed to the BLIP-2 relevance scorer, the correct option is part of the text used to rank tokens. Cosine similarity between a visual token and a phrase like \"blue and gold\" will up-rank the very frame containing evidence for the correct answer. The selected 16 tokens are then passed to the Yi-34B MLLM. This makes the selection procedure answer-aware, not merely query-aware, and could explain a large part of the 4.6-point gap over Qwen2-VL-72B. No ablation in Tables 4/6 or the supplement removes options from q during attention learning, so the reported LVBench numbers are not yet evidence that the representation itself is superior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces SEAL, a representation for long video understanding that decomposes a video into scene, object, and action tokens and then selects a fixed-size subset of these tokens by maximizing an objective that combines query relevance (via BLIP-2 cosine similarity) and token diversity. The selected tokens are passed either to a vision decoder for temporal grounding (Ego4D-NLQ) or to an MLLM for video QA (MovieChat-1K, LVBench). The authors report state-of-the-art results on LVBench with a 34B LLM (45.9% overall, surpassing Qwen2-VL-72B at 41.3%), competitive results on MovieChat-1K, and consistent improvements over SnAG on Ego4D-NLQ under token constraints. The paper includes ablations of the three token types, of the relevance/diversity trade-off, and of global vs. streaming modes.","tokens_in":17559,"tokens_out":4576,"duration_ms":49853,"significance":"If the reported numbers hold, the core idea of semantic decomposition plus query-aware diverse token selection is a useful and practical contribution to long video understanding: it is task-agnostic, supports streaming, and reduces token count substantially. The ablation study isolating each token type is informative, and the partial-observation experiments are a valuable addition. The main caveat is that the headline LVBench result depends on an unresolved specification of what query q is used for relevance scoring; the paper must rule out answer-option leakage before the superiority claim over larger models can be accepted. The paper would be strengthened by reproducible details of the subset-selection optimizer and by error bars on the headline numbers.","major_comments":[{"comment":"The paper never states whether the query q used in the BLIP-2 relevance scoring R(t_s, q) contains the multiple-choice options on LVBench. The LVBench prompt in Section D.2 includes the question and the four options, and Figure 3 displays the options alongside the selected tokens. If the same q is used for relevance scoring, the subset selector becomes answer-aware: tokens matching the correct option text (e.g., 'blue and gold') would be up-ranked, inflating the reported 45.9% overall accuracy. Please state explicitly whether options are stripped from q during attention learning, and run an ablation on LVBench with options removed from q (and with q consisting of the question only) to quantify the effect. This is load-bearing for the claim that SEAL outperforms Qwen2-VL-72B by 4.6%.","section":"Section 3.2 and Section D.2; Figure 3"},{"comment":"The subset-selection objective is presented only as an arg-max problem; no algorithm is given for solving it, nor are its initialization, complexity, or convergence properties discussed. The Limitation paragraph in Section 5 states that Attention Learning is 'bounded by the memory constraint for the QP solver,' but no QP formulation or solver details appear in the paper or supplement. Without this information, the central mechanism is not reproducible and its computational cost cannot be assessed. Please provide the exact optimization procedure (or a reference), including how the pairwise diversity term is handled and how the fixed-size constraint is enforced.","section":"Section 3.2 and Section 5 (Limitation)"},{"comment":"All accuracy numbers are reported as point estimates without error bars, confidence intervals, or repeated-run statistics. The paper itself acknowledges 'high variance in the Summarization accuracy due to the small number of questions in this category and the randomness of LLM.' Given that the headline 45.9% overall accuracy is the basis for claiming state-of-the-art performance, the absence of variability information makes it impossible to determine whether the 4.6-point gap over Qwen2-VL-72B is statistically meaningful. Please report means and standard deviations over at least three runs, or bootstrap confidence intervals, especially for the comparisons in Tables 1 and 4.","section":"Tables 1, 4, and 6; Section 4.3"},{"comment":"The Ego4D-NLQ comparison is made only against SnAG, and the reported gains are very small at the 450-token setting (R@1 0.3: 13.78 vs. 13.44; R@1 0.5: 9.26 vs. 9.23). The paper claims SEAL 'consistently outperforms' SnAG, which is supported in direction, but without statistical significance tests or multiple runs the margin could be noise. Please report variance or significance tests for these numbers, or soften the claim accordingly.","section":"Table 2"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: 'excell' should be 'excel', 'foundamental' should be 'fundamental', 'Attention Leaning' in Section 3.3 should be 'Attention Learning', and 'Moviechat'/'Ego4D-NLP'/'InterVL2' appear alongside 'MovieChat'/'Ego4D-NLQ'/'InternVL2'.","section":"Throughout"},{"comment":"The row labeled 'rep Yolo w/ SAM2' is unclear; the text suggests it refers to replacing YOLO with SAM2 for action-token extraction, but the label and caption should be more explicit.","section":"Table 6"},{"comment":"The caption says 'Cross-Model Fusion' while the text says 'cross-modal fusion' in Section 3.4; please unify the terminology.","section":"Figure 2"},{"comment":"The LVBench prompt shown in Section D.2 includes the options, but it is not stated whether this prompt is also used as q for relevance scoring in Section 3.2; this is related to the major comment and should be clarified directly in that section.","section":"Section D.2"},{"comment":"The sentence 'Scene, action, and object tokens are extracted at 8, 10, 1 FPS for MovieChat-1K, Ego4D-NLQ and LVbench' is ambiguous because the order of datasets does not match the order of the FPS values; please make the mapping explicit.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The answer-option leakage concern in Section 3.2 is the most serious issue. If the authors cannot confirm that options are excluded from q during relevance scoring, the LVBench headline should not be used as evidence of superiority over larger models. The paper is otherwise a reasonable contribution, but it also needs the optimization algorithm details and error-bar reporting before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: SEAL is a sensible engineering contribution for long-video representation, but its headline LVBench number may be inflated by answer-option leakage in the token-selection step. That needs to be resolved before the 45.9% claim is taken at face value.\n\nWhat is actually new: decomposing an hour-long video into scene, object, and action tokens, then choosing a small subset by balancing BLIP-2 relevance to the query against token diversity. Each ingredient comes from prior work (SAM2, trackers, BLIP-2, diversity sampling), but the packaging into one representation that feeds both an MLLM for QA and a vision head for temporal grounding is a new combination that works. The ablations are genuinely informative: the token-type table, the α sweep, and the streaming-window study all tell a clear story. The paper is honest about its own limits, noting that the selection step is not end-to-end trainable and that causal reasoning remains hard.\n\nThe soft spots are real. The biggest is the query q in Eq. (3.2). The paper never states whether, on LVBench's multiple-choice questions, q includes the four answer options. The supplementary shows the MLLM prompt includes options, but the relevance scorer is a separate module. If the correct option's text is part of q, then BLIP-2 cosine similarity will up-rank the very frame that contains the answer, making the selection answer-aware rather than query-aware. That could explain a substantial part of the 4.6-point gap over Qwen2-VL-72B. The omission is easily fixed with an ablation, but as written the headline number is not clean evidence.\n\nSecond, the subset-selection optimization is referred to as a QP solver, but no algorithm, approximation, or complexity is given. That is a reproducibility gap. Third, there are no error bars or multiple seeds anywhere; on Ego4D the gains over SnAG are about 0.3–0.9 points, which is within plausible noise. The MovieChat result is more robust but still single-seed.\n\nThe central idea is plausible and the writing is clear. This is exactly the kind of paper that should go to peer review, with the explicit request that the authors (a) state whether options are stripped from q during attention learning, (b) add an ablation with options removed, (c) specify the solver, and (d) report variance. If they tighten those, the paper could be a solid contribution to the long-video literature.","headline":"Useful integration of semantic tokens and query-aware selection, but the LVBench headline may be inflated by answer-option leakage in the selection step.","tokens_in":18028,"tokens_out":3705,"would_cite":false,"duration_ms":34732,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SEAL claims that a compact set of semantic tokens—scenes, objects, and actions—selected for query relevance and diversity, gives a long-video representation that beats far larger models on QA and temporal grounding.","keywords":["long video understanding","semantic decomposition","attention learning","subset selection","video question answering","temporal grounding","scene object action tokens","streaming representation"],"falsifier":"Re-run SEAL on LVBench with a modified query that contains only the question stem (options removed), and compare the overall accuracy; if accuracy drops materially from 45.9%, the reported gain is inflated by answer leakage through the relevance term.","tokens_in":17152,"feed_emoji":"🎬","tokens_out":4499,"duration_ms":42201,"temperature":0.7,"pith_summary":"SEAL claims that long videos can be reduced to a compact set of semantic tokens—scenes, objects, and actions—and that an attention-learning step that selects a diverse, query-relevant subset of these tokens yields a unified representation strong enough to beat far larger models. On LVBench (hour-long videos), SEAL with a 34B LLM reaches 45.9% overall accuracy, ahead of Qwen2-VL-72B (41.3%) and InternVL2-34B (39.6%). The same representation, with different prediction heads, improves video QA on MovieChat-1K and temporal grounding on Ego4D-NLQ. If this holds, it would show that careful semantic-structured sampling can substitute for raw computation and model scale in long-video understanding.","feed_headline":"Three token types outscore a 72B model on hour-long video QA","feed_subtitle":"Decomposing video into scenes, objects, and actions cuts redundancy and wins on LVBench, MovieChat, and Ego4D.","key_machinery":"The central machinery is the pairing of semantic decomposition with attention learning. Decomposition turns a long video into three token types: scene tokens (uniformly sampled frames), object tokens (SAM/SAM-2 mask crops from key frames), and action tokens (tracklets from a class-agnostic tracker, e.g., YOLOv10-X with BoT-SORT or SAM-2). Attention learning is a subset-selection optimization that balances query relevance $R(\\cdot)$ and token diversity $S(\\cdot)$; the scalar $\\alpha$ controls the trade-off, defaulting to 0.9. This step is what lets the representation drop redundant tokens while keeping content useful for a given query, and it operates in either a global mode (whole video at once) or a streaming mode (sliding window over the selected subset so far).","core_discovery":"The paper's central claim is that temporal redundancy, not model capacity, is the main obstacle to long-video understanding, and that replacing dense frame sampling with a small set of semantic entity tokens—static scenes, static objects, and dynamic actions—plus a subset-selection step that maximizes query relevance while preserving token diversity, yields a representation that transfers across tasks. The subset selection is formulated as optimizing $F_s(T_s|T_G, q) = \\alpha \\sum_{t_s \\in T_s} R(t_s, q) + (1-\\alpha) \\sum_{t_i,t_j \\in T_s, i\\ne j} 1/S(t_i, t_j)$ over candidate token sets, with $R$ computed by BLIP-2 cosine similarity and $S$ by token-pair similarity. SEAL then plugs the selected tokens into either an MLLM head for QA or a vision head with a moment decoder for temporal grounding. The paper reports that this combination outperforms state-of-the-art methods on LVBench, MovieChat-1K, and Ego4D-NLQ, including models with more than twice its LLM size.","pith_inferences":["A testable implication: if answer options are stripped from the query during relevance scoring on LVBench, accuracy should drop; reporting that ablation would clarify how much of the gain depends on query-image matching versus genuine video reasoning.","The subset-selection step is not end-to-end trainable (the paper states a memory constraint on the QP solver); making it differentiable via a continuous relaxation could let the token types and the selection co-adapt with the LLM head, possibly closing part of the remaining gap to human performance.","The same token-decomposition idea could be applied to other modalities, e.g., audio or egocentric sensor streams, wherever redundancy is high and queries are sparse."],"forward_implications":["A unified long-video representation can be shared across video QA, open-ended generative QA, and temporal grounding without task-specific video encoders.","A 34B-LLM pipeline with SEAL tokens can outperform 72B models that process densely sampled frames, suggesting token quality matters more than model scale.","Streaming mode offers a path to arbitrarily long videos: the model updates its representation incrementally and still beats full-video baselines on temporally intensive tasks.","The attention-learning objective's balance between relevance and diversity, with $\\alpha=0.9$, outperforms either term alone, confirming that both focus and coverage matter."],"supporting_citations":[{"why":"Supplies the BLIP-2 encoder used to compute query-token relevance R(·).","marker":"[18]"},{"why":"Provides the SAM-2 tracker used to extract action tracklets and object masks.","marker":"[32]"},{"why":"Contributes the MovieChat dataset and its memory-based baseline; also the evaluation protocol for MovieChat-1K.","marker":"[34]"},{"why":"Defines the SnAG grounding head and IoU/focal losses used for temporal grounding.","marker":"[26]"},{"why":"Supplies the LVBench benchmark, its questions, and the baseline results SEAL is compared against.","marker":"[41]"},{"why":"InternVL2 is the backbone and comparison baseline; its settings are followed on LVBench.","marker":"[3]"},{"why":"Qwen2-VL-72B is the strongest comparison baseline on LVBench.","marker":"[40]"}],"fun_headline_variants":["Scene, object, action tokens beat 72B model on long video QA","Subset-selected scenes, objects, actions beat full-frame baseline","Why few entity tokens beat a 72B model on hour-long video","Attention subset selection trims video redundancy, boosts QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the query q used in the relevance scorer does not leak the correct answer on multiple-choice tasks; on LVBench the query includes the question plus the four answer options, and if those options are not stripped, the token-selection step can partly select tokens simply for resembling the right answer text.","fun_headline_variants_meta":{"raw":{"variants":["Scene, object, action tokens beat 72B model on long video QA","Subset-selected scenes, objects, actions beat full-frame baseline","Why few entity tokens beat a 72B model on hour-long video","Attention subset selection trims video redundancy, boosts QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3658,"prompt_tokens":937,"completion_tokens":2721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":2646}},"tokens_in":553,"tokens_out":2721,"duration_ms":20592,"temperature":1.0,"reasoning_tokens":2646,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:55:39.606807+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run SEAL on LVBench with a modified query that contains only the question stem (options removed), and compare the overall accuracy; if accuracy drops materially from 45.9%, the reported gain is inflated by answer leakage through the relevance term.","supporting_citations":[{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the BLIP-2 encoder used to compute query-token relevance R(·)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the MovieChat dataset and its memory-based baseline; also the evaluation protocol for MovieChat-1K."},{"cited_title":"SnAG: Scalable and accurate video grounding","cited_arxiv_id":null,"evidence_quote":"Defines the SnAG grounding head and IoU/focal losses used for temporal grounding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LVBench benchmark, its questions, and the baseline results SEAL is compared against."}],"review_version":1}