{"id":"52c7d6db-fb68-4ec9-aee7-76868236ff31","arxiv_id":"2411.12355","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"DynFocus dynamically allocates a few tokens to selected frames and two tokens to the rest, reporting competitive video QA accuracy with lower token budgets.","lead":"DynFocus is a method for video question answering that compresses video tokens by using a learned network to pick 'meaningful' frames and encode the rest cheaply. The paper reports competitive results on several video QA benchmarks, but the core claim that frame selection adapts to each question is not actually implemented in the architecture.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed question-adaptive selection is not implemented: Eq. 4 scores frames with U(Max(m_l)||Avg(m_l)), so the DPE mask b is independent of the question, directly contradicting the paper's correspondence motivation and Sec. 3.2's claim of selection 'in accordance with answer and question'.","rationale":"The paper's own framing makes question-adaptive selection load-bearing: the abstract says 'the corresponding frames vary with different questions' and proposes dynamic encoding because of it; Sec. 3.2 concludes that DPE achieves 'dynamic selection in accordance with answer and question.' Equation 4 is the only scoring mechanism for selecting which frames get dense Cones tokens, and its inputs are Max(m_l) and Avg(m_l), both functions of visual prototypes only. Text features Q are introduced only later, in the Rods branch (Eq. 9), after the mask b has already been fixed. Therefore, for a fixed video, b and the top-K set are invariant under changes to the question. The differentiable perturbed top-K (Eqs. 5-7) updates U with LLM gradients, but this can only learn a static importance function over visual content; it cannot make the selector a function of q. This is an internal contradiction with the stated mechanism, not a disagreement with an external consensus. The empirical comparisons, while extensive, do not repair this: the ablations in Table 6 show that some dynamic token allocation helps, but a per-video static importance model would also show such gains. The LV-Bench rows in Table 3 and the B.7 discussion openly select L and K/L on the test benchmark, inflating the reported numbers, and the constraint set in Eq. 5 (P >= 0, 1^T P = 1) permits the same prototype to be selected multiple times, making the differentiable top-K approximation questionable. These are additional reasons not to take the headline numbers at face value, but the question-agnostic selector is the single load-bearing flaw. My concrete test is deliberately minimal: log the selected masks for two questions on the same video. Since U has no question input, the masks must coincide; that observation settles whether the correspondence claim is implemented. I therefore keep the reader's REJECT verdict unchanged.","tokens_in":24675,"tokens_out":4667,"duration_ms":46693,"concrete_test":"Run the released DynFocus code on one video with two different questions whose ground-truth answers depend on disjoint temporal segments (e.g., LV-Bench 'what happens at the beginning?' vs 'what happens at the end?'), and log the DPE index mask b and the top-K selected prototypes for each question. Because Eq. 4 has no question input, the two masks must be identical (apart from nondeterminism). If they are identical, the paper's central 'correspondence' claim is not realized by the architecture and any accuracy gain must be attributed to static per-video importance. If the implementation instead produces different masks, then Eq. 4 is incomplete and the released code should be used to identify the actual question-dependent pathway.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DynFocus dynamically selects answer-relevant frames by exploiting 'correspondence', i.e., that the frames needed vary with the question (abstract, Sec. 1, Fig. 1b). The selection path does not implement this. In Sec. 3.2, Eq. 4 computes s_l = U(Max(m_l)||Avg(m_l)) from pooled visual prototypes only; U has no question argument, and neither the prototypes m_l nor the text features Q enter the score. The binary mask b used in Eq. 11 is derived from the same scores, so for a fixed video the same frames receive fine 'Cones' tokens regardless of the question. The end-to-end LLM gradient mentioned in Sec. 3.2 can make U prefer frames that are useful on average across training questions, but it cannot make the selection question-specific at inference. This is an internal inconsistency, not a matter of consensus: Sec. 3.2 explicitly promises selection 'in accordance with answer and question', and Fig. 5 is presented as showing question-dependent score distributions. If the selectors are identical across questions, the correspondence advantage is absent and the method reduces to a static per-video importance selector, which is a weaker (though possibly still useful) contribution. Secondary concerns—test-set selection of L and K/L in Table 3 and the degenerate top-K constraint in Eq. 5 that permits duplicate selections—further weaken the reported gains, but the missing question conditioning is the load-bearing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DynFocus, a video token-compression pipeline for LLM-based video question answering. A Dynamic Event Prototype Estimation (DPE) module clusters frame features into event prototypes and learns an MLP score to select 'important' prototypes; a Compact Cooperative Encoding (CCE) module then encodes selected frames with fine-grained tokens and the remaining frames with coarse, text-grounded tokens. The method is evaluated on short-video QA benchmarks, three long-video benchmarks, and a video-hallucination benchmark. The paper claims that the selection is question-adaptive and exploits 'correspondence' between frames and questions, but the implementation as written does not condition the selection on the question.","tokens_in":24961,"tokens_out":8313,"duration_ms":88089,"significance":"If the dynamic selection were genuinely question-conditional, DynFocus would offer a useful memory/accuracy trade-off for long-video QA, and the paper is commendable for releasing code, reporting extensive experiments on multiple benchmarks, and including detailed ablations. However, the central mechanism as implemented is not question-conditional, and the evaluation protocol appears to tune hyperparameters on test benchmarks. The descriptive redundancy statistics and the rod/cone analogy are interesting, but they do not compensate for the mismatch between the stated contribution and the actual equations. The paper would need a substantial revision of the selection mechanism and a re-evaluation under a fixed validation protocol before its central claims could be accepted.","major_comments":[{"comment":"The selection score is s_l = U(Max(m_l)||Avg(m_l)), where U is an MLP with no question argument; neither the question features Q nor any text-derived conditioning enters the score. Consequently, for a fixed video, the binary mask b in Eq. (11) is identical for every question, so the claimed 'dynamic selection in accordance with answer and question' is not implemented. The end-to-end LLM gradient can only make U prefer frames that are useful on average over the training questions; it cannot produce question-specific selection at inference. This directly contradicts the correspondence motivation in Sec. 1 and Fig. 1(b), and Fig. 5 does not demonstrate question-dependent selection because it shows different videos for different questions. This is the load-bearing mechanism of the paper and must be addressed, for example by conditioning U on Q or otherwise showing how question information affects b, with a re-evaluation after the change.","section":"Sec. 3.2, Eq. (4)"},{"comment":"The linear-programming formulation of the Top-K selection is degenerate. The preceding text defines P as a stack of one-hot vectors with P ∈ {0,1}^{L×K}, but Eq. (5) states P ∈ R^{K×L} with constraint C = {P : P_{k,l} ≥ 0, 1^T P = 1}. If 1 has length K, then every column of P sums to 1, so the objective ⟨P, s1^T⟩ equals ∑_l s_l for any feasible P, independent of K; the optimization does not select K prototypes. If P is instead L×K as defined earlier, the constraint is dimensionally inconsistent, and when corrected to column-stochastic it permits duplicate selections. The resulting P_σ and H = P^T M therefore do not realize the claimed 'Top-K filtered event prototypes', and the token allocation in Eq. (11) is not as described. A correct differentiable top-K operator with a constraint set that provably selects K distinct prototypes, or a removal of the Top-K claim, is required.","section":"Sec. 3.2, Eqs. (5)-(7)"},{"comment":"The hyperparameters L and K/L appear to be selected per evaluation benchmark. Table 3 reports five DynFocus variants with different (L, K/L) values and marks one as 'optimal results', and App. B.7 states that the optimal L shifts with video length and that a smaller K/L is better for longer videos. Because these choices are made on the test benchmarks and no held-out validation protocol is described, the reported numbers are not a fixed-model comparison and are inflated relative to baselines that use a single configuration. The paper should fix L and K/L using a validation split (or report results across a prespecified grid with appropriate multiple-comparison correction) before claiming competitive or SOTA performance.","section":"Tables 3-4 and App. B.7"}],"minor_comments":[{"comment":"The abstract says 'five publicly available benchmarks', while the contributions in Sec. 1 mention 'two short video benchmarks, three long video benchmarks, and one diagnosis benchmark', which sums to six; please reconcile this count.","section":"Abstract vs. Sec. 1"},{"comment":"The caption contains an undefined superscript '1' after 'video datasets'; the figure should be self-contained or the footnote should be restored, and the redundancy estimation procedure from App. A should be referenced explicitly.","section":"Fig. 1(a)"},{"comment":"The evaluation version of GPT-3.5-Turbo is not fixed across methods; the authors' own note that the default GPT-3.5-Turbo version significantly impacts evaluation makes the comparison ambiguous. Please use one fixed version and state it for all methods.","section":"Table 1"},{"comment":"The notation h_t in Eq. (8) is not defined; the event prototypes are indexed as h_k with k ∈ [1,K], and the mapping from h_k to the frame-specific mask b_t should be clarified.","section":"Eq. (8)"},{"comment":"Reference [13] 'John Doe and Jane Smith. EgoQA' appears to be a placeholder and should be replaced with the actual citation.","section":"References"},{"comment":"The caption for the VideoMME figure mentions 'Medium (K=55)', which suggests per-duration selection of K; if so, this is a test-set selection and should be clearly labeled as such or moved to a validation-based protocol.","section":"App. B.7, Fig."}],"recommendation":"reject","confidential_remarks":"The paper has an extensive experimental section, but the central claimed mechanism — question-adaptive frame selection — is absent from the implementation, and the evaluation protocol selects hyperparameters on the test benchmarks. The linear-programming Top-K formulation also appears mathematically degenerate. I do not see how the current manuscript can be repaired without redesigning the selection module, retraining, and re-running all experiments under a fixed validation protocol; this goes beyond a minor revision. The placeholder reference and benchmark-count inconsistency suggest the manuscript was not fully polished, but my decision is based on the substantive technical issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: the central claim doesn't survive a close read. The paper promises question-adaptive selection of meaningful frames, but the selector in Eq. 4 scores each event prototype with U(Max(m_l)||Avg(m_l))—no question input anywhere. For a fixed video, the binary mask b is identical for every question, so the 'correspondence' motivation in Fig. 1b and the claim in Sec. 3.2 of selection 'in accordance with answer and question' are not implemented. End-to-end training only makes the score network good on average across training questions; it cannot make inference-time selection question-specific.\n\nThat said, the paper has real engineering substance. It combines DPC-KNN clustering, dual-token encoding, a learned importance MLP, and a differentiable perturbed top-k into a token-efficient video encoder. The Cones/Rods cooperative encoding is a useful design pattern. The experimental effort is substantial: short and long video QA, VideoMME, hallucination benchmarks, ablations, latency and parameter tables. Token savings versus LLaMA-VID and ST-LLM look genuine, and the code link is there.\n\nThe soft spots are serious but localized. The missing question conditioning is load-bearing; without it, the method is a static per-video importance selector, a weaker contribution than advertised. Second, hyperparameters L and K/L are tuned on test benchmarks (Table 3, App. B.7), which inflates the reported gains. Minor but real: the top-K linear program constraint in Eq. 5 is mis-specified—1^T P = 1 on a K×L matrix permits duplicate selections, so it doesn't enforce distinct prototypes.\n\nWho's this for? Practitioners working on token-budgeted video LLMs might find the architecture useful as a strong baseline, but researchers should not cite it for question-conditioned selection without checking the gap. I'd send it to peer review because the experiments and engineering justify referee time, but the reviewers should focus on the question-conditioning gap and the test-set tuning. My own verdict would be reject unless the authors reframe the contribution as a static importance-based allocator and fix the top-k constraint.","headline":"The paper's question-adaptive selection is not in the architecture: the selector never sees the question, so the core claim is unsupported despite a solid token-reduction engineering contribution.","tokens_in":25531,"tokens_out":4616,"would_cite":false,"duration_ms":43450,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In long-video question answering, a learned frame selector that spends detailed tokens only on answer-relevant frames can preserve accuracy while cutting memory use sharply.","keywords":["dynamic frame selection","token efficiency","video question answering","long video understanding","density peaks clustering","differentiable top-k","vision-language models","memory-efficient encoding"],"falsifier":"Build a benchmark where the same video is paired with questions whose answers live in disjoint frames (e.g., 'what color is the hat at 00:10?' vs 'what happens at 01:20?'), then compare DynFocus with a variant whose selector also receives the question text. If the question-conditioned variant does not clearly beat the content-only selector — or if an oracle mask computed from ground-truth answer frames does not beat the learned mask — the paper's claimed question-dependent correspondence is not what drives the results.","tokens_in":24427,"feed_emoji":"🎬","tokens_out":10798,"duration_ms":99664,"temperature":0.7,"pith_summary":"DynFocus claims that video question answering can be made memory-efficient by treating frames asymmetrically: spend many tokens on frames that carry the answer, and compress the rest to a sketchy token pair. The paper supports this with a statistical observation that long videos contain many repeated or answer-irrelevant frames, and that which frames are relevant shifts with the question. It then builds a two-module network — DPE selects meaningful frames and CCE encodes selected frames richly and unselected frames coarsely — and reports competitive or better accuracy than prior video LLMs on several QA benchmarks while using far fewer tokens. If the claim holds, token allocation in video LLMs becomes a learnable decision rather than a fixed uniform cost.","feed_headline":"DynFocus cuts video-QA tokens by focusing on the frames that matter","feed_subtitle":"A learned frame selector keeps video-QA accuracy while using far fewer tokens.","key_machinery":"The load-bearing mechanism is a learned frame-importance gate. Temporal DPC-KNN clustering produces $L$ event prototypes from pooled frame features; a small MLP score network $U$ assigns each prototype an importance score from its max- and average-pooled features ($s_l = U(\\mathrm{Max}(m_l)\\|\\mathrm{Avg}(m_l))$); and a differentiable perturbed-top-K operator converts those scores into a binary mask $b_t$ over frames. The mask drives a cooperative encoder: selected frames receive a concatenation of event and multi-grained spatial prototypes as detailed tokens, while unselected frames are compressed to two tokens via average pooling plus text-guided attention. The token-allocation rule $O_t = b_t\\cdot(U_{t,b_t=1}\\|U_{t,b_t=0})+(1-b_t)\\cdot U_{t,b_t=0}$ is the identity that carries the argument: it makes the token count a learned, content-dependent quantity rather than a fixed per-frame constant.","core_discovery":"The central claim is that dynamic, content-dependent token allocation can preserve question-answering accuracy under a tight token budget. Concretely, DynFocus clusters frame features temporally into event prototypes, learns a score function $U$ that ranks them by answer relevance via $s_l = U(\\mathrm{Max}(m_l)\\|\\mathrm{Avg}(m_l))$, and then allocates fine-grained tokens to the selected frames and two compact tokens to every other frame, per $O_t = b_t\\cdot(U_{t,b_t=1}\\|U_{t,b_t=0}) + (1-b_t)\\cdot U_{t,b_t=0}$. Selected frames are encoded with multi-grained spatial prototypes; the rest are reduced to a text-grounded global token plus a content token, providing a coarse temporal view. Trained end-to-end with a differentiable perturbed top-K, the model reaches competitive or state-of-the-art accuracy among the compared open baselines on short-video QA, MLVU, LV-Bench, and VideoMME, and places second among compared open models on the VideoHallucer hallucination diagnosis.","pith_inferences":["The selector never sees the question text (Eq. 4 takes only prototype features), so the reported 'correspondence' may be an emergent property mimicked by content saliency rather than true per-question adaptation; conditioning the selector on the question is the natural next experiment and a sharper test of the paper's motivation.","The rod/cone split suggests a general principle for multimodal token budgets: a cheap perceptual stream can provide temporal or global context while a costly detailed stream focuses on the answer-relevant region; the same allocation idea could transfer to audio or 3D video.","Supervising the selector only with the final answer may under-specify which frames to keep; auxiliary objectives such as temporal grounding or frame-relevance labels could stabilize selection and make the masks interpretable.","The paper's own ablation shows that removing the coarse stream costs several accuracy points even though it contains only two tokens per frame, so the coarse stream is supplying temporal continuity rather than acting as mere compression; token diversity, not just token count, appears to drive the gains."],"forward_implications":["Long-video QA becomes feasible under fixed memory: on LV-Bench, DynFocus processes 200 frames with a 7B model and beats open baselines that ingest thousands of frames.","The token budget becomes a tunable dial: increasing the number of prototypes $L$ improves accuracy up to a point, and lowering the selection ratio $K/L$ helps on longer videos, so practitioners can trade memory for accuracy per domain.","Frame selection can be trained from the LLM's own response loss without extra frame-level labels, because the perturbed top-K keeps the selector end-to-end differentiable.","Filtering answer-irrelevant frames reduces visual noise, which the paper connects to better factual correctness scores on VCG-Bench and competitive hallucination diagnoses on VideoHallucer."],"supporting_citations":[{"why":"LLaMA-VID's two-token-per-frame encoding and two-stage training recipe give DynFocus its baseline and budget comparison.","marker":"[35]"},{"why":"Chat-UniVi supplies the DPC-KNN clustering pipeline that DynFocus adapts for temporal and spatial prototypes.","marker":"[25]"},{"why":"The density-peaks clustering algorithm defines the local density and distance indicators used to estimate event prototypes.","marker":"[14]"},{"why":"Perturbed differentiable optimizers make the top-K prototype selection end-to-end trainable through the LLM loss.","marker":"[6]"},{"why":"InstructBLIP provides the pre-trained Q-Former text encoder used to produce question embeddings.","marker":"[12]"},{"why":"EVA-CLIP supplies the frozen ViT-G/14 visual encoder that produces frame features.","marker":"[15]"},{"why":"ST-LLM is the strongest short-video baseline that DynFocus matches or slightly exceeds on MSVD-QA and MSRVTT-QA.","marker":"[43]"},{"why":"VideoChat2 is the widely used 7B video LLM baseline that DynFocus outperforms on VideoMME and MLVU while sharing training data.","marker":"[32]"}],"fun_headline_variants":["DynFocus picks the frames that answer, cutting video-QA tokens","Event prototypes guide DynFocus to spend tokens only where it counts","DynFocus slashes tokens by focusing on answer-relevant frames","Video QA gets lean: DynFocus allocates tokens to meaningful frames","DynFocus: dynamic frame selection preserves accuracy, trims token budget"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The selection score is computed from frame content alone, with no input from the question, so the method assumes that one learned importance function can identify the frames that matter for any question.","fun_headline_variants_meta":{"raw":{"variants":["DynFocus picks the frames that answer, cutting video-QA tokens","Event prototypes guide DynFocus to spend tokens only where it counts","DynFocus slashes tokens by focusing on answer-relevant frames","Video QA gets lean: DynFocus allocates tokens to meaningful frames","DynFocus: dynamic frame selection preserves accuracy, trims token budget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2857,"prompt_tokens":929,"completion_tokens":1928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1836}},"tokens_in":545,"tokens_out":1928,"duration_ms":13527,"temperature":1.0,"reasoning_tokens":1836,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:37:10.410576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a benchmark where the same video is paired with questions whose answers live in disjoint frames (e.g., 'what color is the hat at 00:10?' vs 'what happens at 01:20?'), then compare DynFocus with a variant whose selector also receives the question text. If the question-conditioned variant does not clearly beat the content-only selector — or if an oracle mask computed from ground-truth answer frames does not beat the learned mask — the paper's claimed question-dependent correspondence is not what drives the results.","supporting_citations":[{"cited_title":"Study on density peaks clustering based on k-nearest neighbors and principal component analysis","cited_arxiv_id":null,"evidence_quote":"The density-peaks clustering algorithm defines the local density and distance indicators used to estimate event prototypes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"InstructBLIP provides the pre-trained Q-Former text encoder used to produce question embeddings."},{"cited_title":"EV A: exploring the limits of masked visual representation learning at scale","cited_arxiv_id":null,"evidence_quote":"EVA-CLIP supplies the frozen ViT-G/14 visual encoder that produces frame features."}],"review_version":1}