{"id":"eeb8c84e-e92c-44ec-8c7f-48f4948b355d","arxiv_id":"2412.09919","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"B-VLLM selects question-relevant frames and spatial tokens before feeding a video to a vision LLM, improving video QA accuracy under a fixed token budget.","lead":"B-VLLM is a video understanding system that cuts the number of visual tokens fed into a large language model by selecting the most relevant frames and the most relevant parts of those frames, guided by the question being asked. It reports large gains on video QA benchmarks while using fewer tokens, though some of the headline claims are stronger than the tables support.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frame selection uses CLS tokens, which the paper's own Table 7 shows is less informative than mean-pooled or Q-Former features on multiple benchmarks; this weakens the core mechanism but does not refute the empirical gains.","rationale":"The reader identified the same weakest assumption: CLS tokens may not carry enough task-relevant information for frame selection. The paper's own supplementary Table 7 strengthens this concern by showing that mean pooling and Q-Former features outperform CLS on several benchmarks, including spatially demanding OCR and Counting. However, this concern does not refute the central empirical claim: B-VLLM with CLS-based selection still shows consistent gains over LLaMA-VID and VideoLLaMA2 in the main tables and ablations. The concern affects the robustness and upper bound of the method, not its basic validity, so the reader's CONDITIONAL verdict remains appropriate. I also considered the Gumbel-Softmax inconsistency and the overstated SOTA claim, but those are more about presentation and would be addressable without changing the core result. The CLS limitation is more load-bearing because it sits at the first and most consequential selection stage, and it is explicitly acknowledged in the manuscript.","tokens_in":550,"tokens_out":4281,"duration_ms":61848,"concrete_test":"Retrain B-VLLM with the frame-selection module using mean-pooled visual tokens instead of CLS tokens, keeping all other modules, training data, token budgets, and evaluation settings identical. Compare on MVBench, VideoMME (especially the OCR and Counting subsets), and EgoSchema. If mean-pooled selection improves results, the CLS token is confirmed as a bottleneck and the reported gains are conservative; if it does not, the CLS choice is not the limiting factor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that a single CLS token per frame preserves enough question-relevant information for the frame-selection module (Sec. 3.2) to identify the correct frames. The paper itself notes this limitation (Sec. 5.4), and its supplementary Table 7 provides direct evidence: using mean-pooled visual tokens as frame features outperforms CLS on MVBench (51.3 vs 50.8), VideoMME (54.4 vs 52.9), VideoMME-OCR (48.2 vs 46.0), and VideoMME-Counting (36.6 vs 34.0), while a Q-Former feature also outperforms CLS on MVBench and EgoSchema. Since OCR and counting are precisely tasks requiring fine-grained spatial details that may reside in non-CLS tokens, the frame selector can discard the frames needed to answer such questions. This does not invalidate the central claim that the framework yields gains over LLaMA-VID and VideoLLaMA2, but it means the mechanism is knowingly built on a suboptimal representation. The paper justifies CLS by training efficiency (10.9h vs 13.1h in Table 7), which is a practical trade-off, not a correctness argument. A more informative selection feature could change the magnitude and possibly the ranking of reported results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces B-VLLM, a framework for video large language models that controls the number of visual tokens by combining text-conditioned frame selection (using per-frame CLS tokens and a Q-Former), temporal frame-token merging to remove duplicates, spatial token sampling via another Q-Former, and an optional iterative token-merging step. The method is evaluated on several video QA benchmarks and two image benchmarks, both as a standalone model (with Qwen2) and as an integration on top of LLaMA-VID and VideoLLaMA2. The authors report consistent gains over the two baselines, claim state-of-the-art performance on several benchmarks, and provide ablations for each module and for the main hyperparameters. The paper also includes a transparent limitations section and a supplementary study comparing frame-selection features.","tokens_in":19022,"tokens_out":9034,"duration_ms":91345,"significance":"If the reported results are reproducible, B-VLLM is a practical contribution to efficient long-video understanding: it keeps the visual-token budget fixed and shows that text-conditioned selection and spatial sampling can outperform uniform frame sampling and per-frame compression baselines on several benchmarks. The paper's strengths include a clear architecture description, an extensive set of ablations (Tables 3-4, Figure 3, and supplementary tables), a code release, and a fair comparison of training data amounts. The main weakness is that the central frame-selection mechanism relies on CLS tokens that the authors' own supplementary ablation shows to be inferior to mean pooling, and the unqualified state-of-the-art claim conflicts with entries in the paper's own tables. These issues, together with the use of the test benchmarks for hyperparameter selection, require attention before the central claims can be accepted as stated.","major_comments":[{"comment":"The claim that B-VLLM 'achieves SOTA performance' is contradicted by the paper's own results: in Table 1, VideoChat2 (51.1) and ShareGPT4Video (51.2) outperform B-VLLM (50.8) on MVBench, and Supplementary Table 9 shows Qwen2VL, LLaVA-OV, and InternVL2 well above B-VLLM on VideoMME and EgoSchema. The abstract and introduction repeat the SOTA claim without the qualification that it holds only with respect to the same-training-data baselines. Please scope the claim to the actual comparison set and ensure the headline does not overstate the contribution.","section":"§5.1, Table 1 and Supplementary Table 9"},{"comment":"The frame-selection module uses CLS tokens for efficiency, but the paper's own ablation (Table 7) shows that mean pooling gives 51.3 vs 50.8 on MVBench, 54.4 vs 52.9 on VMME, 48.2 vs 46.0 on VMME-OCR, and 36.6 vs 34.0 on VMME-Counting; a Q-Former feature also improves MVBench and EgoSchema. The statement in §5.3 that CLS is 'sufficient' for frame selection is not supported by these numbers. Since adaptive frame selection is a core contribution, please either adopt the better feature in the final model, report main results with the best feature, or explicitly position the reported model as an efficiency-optimized variant and state the performance cost.","section":"§5.3 and Supplementary Table 7"},{"comment":"The hyperparameters L*, R, τ, γ, and θ are analyzed by sweeping on MVBench and VideoMME-Medium, which are the same benchmarks used for the final reported accuracies. This is a form of test-set leakage and the paper's characterization of the evaluation as 'zero-shot' is therefore overstated. Please select hyperparameters on a held-out validation split, or report the tuning protocol explicitly and discuss the risk of overfitting.","section":"Figure 3"}],"minor_comments":[{"comment":"The notation V* = Sτ·V denotes a soft convex combination, not a hard selection, yet the text refers to 'selected frames' throughout. Please clarify how hard selection is implemented at inference (for example, by taking the argmax row of Sτ or by setting τ to a very small value) and how this interacts with the temporal-order restoration.","section":"§3.2, Eq. (1)"},{"comment":"Averaging visual-token sets of duplicate frames is not well defined; please specify whether the average is element-wise over spatially aligned tokens or performed using a more invariant pooling scheme.","section":"§3.3, Eq. (4)"},{"comment":"The -1.7 drop on VideoMME-Short for VideoLLaMA2 w. Ours is not mentioned in the text, which claims consistent short-video gains; please acknowledge this exception in the discussion.","section":"§5.1, Table 1"},{"comment":"The phrase '10% performance gain on MVBench' is ambiguous (absolute vs. relative) and does not match the numbers in Table 1; please specify the baseline and the type of gain.","section":"Abstract and Introduction"},{"comment":"The plots lack axis labels and benchmark legends, so readers cannot determine which curve corresponds to which benchmark or what the plotted quantity is.","section":"Figure 3"},{"comment":"Please report the hardware and software configuration used for the training-time comparison so that the efficiency claim in §5.3 is reproducible.","section":"Supplementary Table 7"},{"comment":"The claim of being 'for the first time' in iterative token merging for controllable token counts is not substantiated; please soften the statement or provide a more specific comparison with prior controllable token-reduction methods.","section":"§2.2"},{"comment":"There are duplicated citations (for example, MME appears as both [13] and [14]), and the reference to 'EV A-CLIP' in §7.4 appears to be missing its arXiv identifier.","section":"References and Supplementary §7.4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core, but the unqualified SOTA claim, the test-set hyperparameter tuning, and the suboptimal CLS selection feature are the main risks. If the authors adopt a held-out validation protocol and rescope the claims, the work could be a reasonable fit for the journal; as it stands, the verification of the central claims requires additional work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"B-VLLM is a genuinely useful integration: text-conditioned frame selection, temporal de-duplication, and spatial token sampling and merging under a fixed token budget. The individual pieces are all prior work, but the combination is new and the experiments support the main mechanism. The gains over LLaMA-VID and VideoLLaMA2 are consistent across most benchmarks, the ablations isolate each module's contribution, and the paper is admirably transparent about its limitations, including the CLS-token concern and the fixed L*. Code is released, training data is specified, and the complexity numbers are reported. That is real evidence and it is treated as such. The soft spots are real but addressable. The SOTA claim is overstated relative to Table 1: B-VLLM does not beat VideoChat2 on MVBench or Qwen2-VL on several benchmarks, and the supplementary comparison shows clear gaps to models trained on more data. The paper should either scope the claim to same-training-data comparisons or drop it. There are no error bars anywhere, and hyperparameters (tau, gamma, L*, R, theta) are tuned on the same benchmarks used for final results. That is selection bias, not circularity, but it should be stated and ideally re-evaluated on a held-out split. The CLS-token concern from the stress-test is legitimate and worth flagging. Table 7 shows mean pooling, max pooling, and Q-Former features all beat CLS on several benchmarks, especially OCR and counting. The paper acknowledges this and justifies CLS by training efficiency. That is a reasonable practical trade-off, but it does mean the frame selector is knowingly built on a weaker representation, and the frame-selection module is the load-bearing component of the whole framework. The soft Gumbel-Softmax story in Eq. 1 also sits uneasily with the discrete frame-selection narrative; the paper should clarify whether selection is hard or soft at inference and whether the row-wise similarity threshold is applied to the relaxed or the discretized matrix. None of this refutes the central empirical claim: under a fixed token budget, question-conditioned selection beats uniform sampling and extreme per-frame compression. That is a subfield-relevant result for video QA practitioners. The paper deserves a serious referee, and with the SOTA claim scoped, error bars added, and the CLS limitation discussed in the main text rather than only in the supplement, it would be a solid accept. For my own work, I would not build on the CLS selector directly, but I would cite the framework as a point of comparison for token-budget control in video LLMs.","headline":"Solid, well-executed engineering paper on balancing spatio-temporal visual tokens for video LLMs; the central claim holds, but the SOTA framing and CLS-based frame selection need tighter handling.","tokens_in":776,"tokens_out":833,"would_cite":true,"duration_ms":19482,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A text question can pick the video frames that matter, keeping long-video QA accurate on a fixed token budget.","keywords":["vision large language model","video question answering","long-video understanding","visual token budget","adaptive frame selection","token merging","CLS token","Gumbel-Softmax"],"falsifier":"Take a video in which the answer appears in a single brief frame whose global CLS token is dominated by irrelevant content, and ask a question that targets that frame; if a variant that selects frames from pooled visual tokens answers correctly while the CLS-based selector does not, the load-bearing shortcut is falsified.","tokens_in":18557,"feed_emoji":"🎥","tokens_out":4177,"duration_ms":44470,"temperature":0.7,"pith_summary":"The paper argues that the right way to fit long videos into a vision-language model's context window is not to choose between temporal and spatial cues, but to spend a fixed visual-token budget on both. It proposes a framework in which a text-conditioned module first selects the frames most relevant to the question, de-duplicates them, then samples and merges the most informative visual tokens inside each selected frame. On video benchmarks spanning ten seconds to one hour, this balanced allocation consistently beats the two existing strategies: uniform frame sampling and heavy per-frame compression. If the claim holds, long-video question answering stops being a trade-off between seeing enough frames and seeing enough detail.","feed_headline":"Long-video QA stays accurate when the question picks the frames","feed_subtitle":"It selects question-relevant frames, then compresses each frame's tokens to fit the context window.","key_machinery":"The mechanism is a two-stage token allocator: a Gumbel-Softmax frame-selection network (a differentiable approximation to discrete sampling) turns the text question and [CLS] tokens into a sparse matrix $S_\\tau$ over frames, followed by a spatial Q-Former (a lightweight transformer that pools many tokens into a few query tokens) that samples $R$ question-relevant tokens per selected frame, with optional bipartite token merging to meet a hard token ceiling $\\theta$. The [CLS] token is the load-bearing shortcut: it makes whole-video frame selection cheap, so the budget can be spent on fine tokens only where they matter.","core_discovery":"The central discovery is that a single visual-token budget can support both temporal and spatial understanding if the allocation is conditioned on the text prompt. Each frame contributes a coarse [CLS] token and a set of fine visual tokens; the [CLS] tokens are cheap enough to let a selection network scan all frames, and the selected frames' fine tokens are then reduced by sampling and optional iterative merging to hit the budget. In the paper's experiments, this yields consistent gains over both a baseline that compresses each frame to two tokens and a baseline that uniformly samples eight frames, with the largest gains on long-video benchmarks and a reported ten-percent gain on one multi-purpose video benchmark.","pith_inferences":["A natural next test would feed the model 'needle' videos where the answer lives in a single brief frame whose CLS token is dominated by irrelevant content; such cases would reveal whether coarse CLS semantics suffice or whether the selection stage needs pooled features.","The same text-conditioned allocation could apply to other long-input settings, such as document retrieval, where a cheap global descriptor filters before expensive local tokens are spent.","The paper's saturation result around 512 visual tokens suggests a practical rule of thumb: most video question answering can be answered from roughly half a thousand visual tokens if selection is question-conditional.","The paper itself points toward removing the fixed selected-frame count $L^*$; an adaptive count would let extremely long videos with many relevant moments avoid discarding information."],"forward_implications":["Long-video benchmarks that previously required hour-long context windows can be answered with a fixed, modest number of visual tokens.","Text-conditioned frame selection can outperform uniform frame sampling, so a video model no longer needs to see every frame to reason about the whole video.","Heavy per-frame compression (two tokens per frame) is not necessary for temporal reasoning; keeping more spatial tokens on selected frames improves both spatial and temporal benchmarks.","The same token-sampling machinery transfers to image benchmarks, where the model performs comparably to image-only models while using far fewer visual tokens per image.","Because the token budget can be set in advance, with performance plateauing rather than degrading, accuracy and compute can be traded deliberately."],"supporting_citations":[{"why":"Supplies the two-tokens-per-frame baseline whose extreme spatial compression the proposed method replaces.","marker":"[30]"},{"why":"Supplies the uniform eight-frame sampling baseline whose temporal sampling the proposed method replaces.","marker":"[9]"},{"why":"Supplies the Q-Former architecture reused for both frame selection and spatial token sampling.","marker":"[26]"},{"why":"Supplies the bipartite token-merging technique adapted into the iterative spatial merging strategy.","marker":"[7]"},{"why":"Provides MVBench, the benchmark used for the headline performance gain and the central ablations.","marker":"[27]"},{"why":"Provides VideoMME, the short/medium/long video benchmark that measures the long-video gains.","marker":"[15]"}],"fun_headline_variants":["Question picks frames to balance video tokens","Let the prompt choose frames for efficient long-video QA","Text-guided frame selection trims token load for long videos","B-VLLM: ask the question, balance spatio-temporal tokens","Efficient long-video understanding via prompt-based frame selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The frame-selection stage trusts a single coarse summary token per frame; if the detail a question depends on survives only in the frame's other visual tokens, that frame can be discarded before it is ever examined.","fun_headline_variants_meta":{"raw":{"variants":["Question picks frames to balance video tokens","Let the prompt choose frames for efficient long-video QA","Text-guided frame selection trims token load for long videos","B-VLLM: ask the question, balance spatio-temporal tokens","Efficient long-video understanding via prompt-based frame selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00067,"raw_usage":{"total_tokens":3078,"prompt_tokens":995,"completion_tokens":2083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2003}},"tokens_in":611,"tokens_out":2083,"duration_ms":15875,"temperature":1.0,"reasoning_tokens":2003,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:34:15.429949+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video in which the answer appears in a single brief frame whose global CLS token is dominated by irrelevant content, and ask a question that targets that frame; if a variant that selects frames from pooled visual tokens answers correctly while the CLS-based selector does not, the load-bearing shortcut is falsified.","supporting_citations":[{"cited_title":"LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models","cited_arxiv_id":null,"evidence_quote":"Supplies the two-tokens-per-frame baseline whose extreme spatial compression the proposed method replaces."},{"cited_title":"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models","cited_arxiv_id":null,"evidence_quote":"Supplies the Q-Former architecture reused for both frame selection and spatial token sampling."},{"cited_title":"Token Merging: Your ViT But Faster","cited_arxiv_id":null,"evidence_quote":"Supplies the bipartite token-merging technique adapted into the iterative spatial merging strategy."},{"cited_title":"MVBench: A Comprehensive Multi-Modal Video Under- standing Benchmark","cited_arxiv_id":null,"evidence_quote":"Provides MVBench, the benchmark used for the headline performance gain and the central ablations."}],"review_version":1}