{"id":"b1d0ed26-4f21-4c6f-8b3e-83ce299cc420","arxiv_id":"2506.22139","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A CLIP-scored, Gumbel-Max frame sampler with per-frame multi-resolution allocation improves long-video question answering in Video-LLMs under a fixed token budget.","lead":"Q-Frame is a training-free method that picks video frames relevant to the question and assigns each frame a different resolution, so video language models can see more useful frames within a fixed token budget. It reports accuracy gains over uniform sampling on long-video benchmarks including MLVU, LongVideoBench, and Video-MME.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Appendix B.5 concedes that Q-Frame discards sequential structure and causal relationships; the 'consistently outperforms uniform sampling' claim therefore cannot hold for temporal-reasoning tasks, and Table 9 already shows negative deltas on temporal subtasks.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing mechanism: Q-Frame selects frames by static per-frame CLIP similarity and cannot preserve order or transitions. I agree with that identification. The paper itself contains the decisive concession in Appendix B.5, so this is not an external speculation about a hypothetical failure mode; it is a documented limitation of the proposed selection process. Since video QA includes temporal reasoning where event order and boundaries matter, a per-frame selector can omit precisely the information needed. Table 9 provides internal evidence: several temporal subtask accuracies decrease relative to uniform sampling, contradicting the word 'consistently' in the central claim. The evaluation-related concerns raised by the reader (the selected GPT-4o MLVU baseline, hyperparameter tuning on the reported LongVideoBench set, and missing error bars) are real but secondary: they affect the precision of the headline gains rather than the mechanism itself. The cleanest resolution is to narrow the central claim to tasks where query-relevant static frames suffice, and to add a dedicated temporal-reasoning evaluation. Because the reader already assigned CONDITIONAL, my read does not move the verdict; it strengthens the condition under which the paper should be accepted.","tokens_in":16882,"tokens_out":4608,"duration_ms":51393,"concrete_test":"On the temporal-reasoning categories of LongVideoBench (aggregating TOS, T3O, T3E, T2O, T2E, TAA, T2A, SOS) and on Video-MME's temporal Reasoning questions, run Q-Frame and the uniform baseline under the same token budget from Eq. (8), with at least 5 independent Gumbel draws per video, and report mean +/- std. If the mean Q-Frame delta over uniform is not positive on this temporal subset, the 'consistently outperforms' claim must be restricted to non-temporal tasks; if it is positive, the Appendix B.5 bad case is an outlier and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that per-frame query-relevance scoring is sufficient to select the frames a Video-LLM needs, including for temporal reasoning. Section 3.3 introduces Gumbel noise precisely because 'CLIP-like models are primarily trained on image-text pairs, which inherently neglect the temporal correlations present in video frames.' Appendix B.5 then concedes that Q-Frame 'doesn't preserve the sequential structure or causal relationships between events that are critical for effective temporal reasoning.' This is not a peripheral caveat: Table 9 (LongVideoBench subtasks) contains multiple negative entries on temporal categories, e.g., VILA-V1.5 drops from 53.0 to 40.9 on T3O and from 41.1 to 34.2 on T3E, and Qwen2-VL drops from 78.5 to 70.8 on T2E. Thus the abstract's 'consistently outperforms uniform frame sampling' and 'preserving critical temporal and spatial information' are internally under-supported. The method may still improve factoid, spatial, and OCR-style tasks, but the broad claim must be narrowed unless temporal-reasoning performance is separately demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Q-Frame, a training-free, plug-and-play frame-selection and multi-resolution adaptation method for Video-LLMs. It uses CLIP-style image-text similarity to score uniformly sampled candidate frames, adds Gumbel noise, selects top-K frames, and allocates different resolutions (high/medium/low) to preserve fine details while fitting a fixed token budget. The method is evaluated on MLVU, LongVideoBench, and Video-MME with VILA-V1.5, GPT-4o, and Qwen2-VL backbones. The headline results show consistent gains for Qwen2-VL (e.g., MLVU 56.9→65.4, LongVideoBench 53.5→58.4, Video-MME without subtitles 53.7→58.3) and for VILA-V1.5 on MLVU and LongVideoBench, while GPT-4o gains are small and the MLVU baseline is unstable across runs. The paper includes ablations on QFS vs. MRA, resolution allocation, temperature, CLIP backbone, and candidate-frame count.","tokens_in":17199,"tokens_out":3253,"duration_ms":33735,"significance":"If the reported gains hold, Q-Frame would be a practically valuable addition to Video-LLM inference: it is model-agnostic, requires no training, and its ablations show that query-relevance-based selection alone improves over uniform sampling at fixed resolution (Table 4). The multi-resolution ablation (Table 5) and the overhead measurements (Table 14) are useful contributions. The paper also provides detailed per-subtask results, which are commendable. However, the central claim that Q-Frame 'preserv[es] critical temporal and spatial information' and 'consistently outperforms uniform frame sampling' is contradicted by the paper's own limitation statement and per-subtask temporal results; the GPT-4o MLVU evidence is within run-to-run noise; and hyperparameters appear to be selected on the same benchmark used for reporting. The method may still be valuable for query-dependent, spatially grounded tasks, but the current claims overreach.","major_comments":[{"comment":"The abstract claims Q-Frame 'preserv[es] critical temporal and spatial information' and 'consistently outperforms uniform frame sampling.' Appendix B.5 explicitly concedes that the selection process 'doesn't preserve the sequential structure or causal relationships between events that are critical for effective temporal reasoning.' Table 9 shows multiple negative deltas on temporal subtasks, including VILA-V1.5 dropping from 53.0 to 40.9 on T3O and from 41.1 to 34.2 on T3E, and Qwen2-VL dropping from 78.5 to 70.8 on T2E. This is an internal inconsistency between the stated contribution and the evidence. Please narrow the central claim to tasks where per-frame relevance is sufficient, or provide separate evidence that temporal-reasoning performance is preserved (e.g., with temporal-aware selection).","section":"Abstract and Appendix B.5; Table 9"},{"comment":"The GPT-4o MLVU result is not supported by the data. Appendix B.2 reports three baseline runs of 27.4, 28.6, and 28.3, and a Q-Frame run of 29.3; the manuscript reports 28.6 as the baseline, yielding a 0.7-point gap that is smaller than the baseline's own run-to-run spread of 1.2 points. This difference is within noise and cannot support the claim that Q-Frame improves GPT-4o on MLVU. Please either provide statistically meaningful evidence (multiple Q-Frame runs with error bars and a significance test) or remove/reframe this specific claim.","section":"Appendix B.2 and Table 2"},{"comment":"Hyperparameters including temperature tau (Table 13), resolution allocation K/M/N (Table 6), the CLIP-like backbone (Table 12), and candidate-frame count T (Table 14) are tuned by peeking at LongVideoBench and then reported on the same benchmark. Since no held-out validation is described, the reported gains may be inflated by selection on the test set. Please evaluate the method on a validation split or, failing that, report sensitivity across a range of hyperparameters and state explicitly that the reported configuration was chosen on the benchmark itself; the 'consistently outperforms' claim should then be tempered accordingly.","section":"Section 4.3 and Appendix B.4"},{"comment":"The token-budget equation K + M/4 + N/16 = 8 assumes that visual-token consumption scales exactly with the linear resolution ratio r(1):r(2):r(3) = 1:2:4. Table 6 shows that actual tokens/video vary from 2265.1 to 2370.1, a 4.6% range, so the equation is at best approximate. More importantly, Qwen2-VL's native dynamic tokenization may not follow a simple area-law scaling. Please report actual token counts and inference latency for the configurations in Table 6, and justify that the approximation is tight enough to support the claim that Q-Frame 'process more frames without exceeding computational limits.'","section":"Section 3.4, Eq. (8)"}],"minor_comments":[{"comment":"The table numbering is inconsistent: the repeated GPT-4o experiment is referred to as 'Table 10' in the text, but the presented tables are labeled Table 8 (parameter config), Table 8 (repeated runs), Table 9 (subtasks), and Table 10 (MLVU subtasks). Please renumber consistently and fix the in-text references.","section":"Appendix B.2"},{"comment":"The duration ranges '(8s, 15s]' and '(15s, 1m]' skip the interval (1m, 3m]; please clarify whether the (1m, 3m] category was omitted or merged.","section":"Table 3"},{"comment":"The bar chart for Video-MME task accuracies is difficult to read because the baseline and Q-Frame values are close and the y-axis labels are not visible. Please add value labels and a legend.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is simple and potentially useful, and the Qwen2-VL and VILA-V1.5 results support a narrower claim about query-relevant frame selection for spatially grounded tasks. However, the temporal-reasoning concession in Appendix B.5 directly undercuts the abstract's 'preserving critical temporal and spatial information' claim, and the GPT-4o MLVU evidence is within noise. With careful rewording and additional validation, a revised version could be acceptable. The hyperparameter selection on the same benchmark should also be disclosed or addressed, as it affects how readers interpret the reported gains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Q-Frame is a training-free plug-in for video-LLMs that selects query-relevant frames via CLIP similarity plus Gumbel noise and then allocates different resolutions to those frames under a fixed token budget. That combination is genuinely new even though each component is known. The paper gives credit where due: the ablations show query-based selection helps at fixed resolution, the resolution-allocation study is reasonably thorough, and they include a limitations section plus a bad-case analysis, which is more honest than most.\n\nThe soft spots are real, and two of them are load-bearing. First, the GPT-4o MLVU result in Table 2 should be retracted. Appendix B.2 reports three baseline runs (27.4, 28.6, 28.3) and one Q-Frame run (29.3); the paper adopts the best baseline (28.6). That gap is within run-to-run noise and the adoption is post hoc. Second, hyperparameters (tau, K/M/N, CLIP backbone) were tuned on LongVideoBench and then reported on that same set, so the 58.4 headline there is optimistic. There are no error bars anywhere, so the Qwen2-VL gains on MLVU and Video-MME cannot be assessed for significance either.\n\nThe more substantive concern is the temporal-reasoning claim. The abstract says Q-Frame 'consistently outperforms uniform frame sampling' and preserves 'critical temporal and spatial information.' But Appendix B.5 concedes the selection process doesn't preserve sequential structure or causal relationships, and Table 9 shows negative deltas on several temporal subtasks (VILA-V1.5 T3O drops 53.0 to 40.9, T3E 41.1 to 34.2; Qwen2-VL T2E 78.5 to 70.8). That doesn't kill the method—it can still help factoid, spatial, OCR, and counting tasks—but the broad claim has to be narrowed, or the paper needs separate evidence that temporal reasoning is not hurt.\n\nWhere does that leave the paper? It is a legitimate contribution to the frame-selection subfield: simple, training-free, and likely useful as a baseline for video-LLM work. The Qwen2-VL results are suggestive even if not yet statistically grounded. It deserves a serious referee, but only after the authors fix the GPT-4o analysis, add error bars or repeated runs, and adjust the abstract to match the temporal caveat. I'd bring it to a reading group as an example of both a useful plug-in and a set of evaluation pitfalls.","headline":"Q-Frame is a plausible training-free frame-selection plug-in with solid Qwen2-VL gains, but the GPT-4o MLVU result is noise, the LongVideoBench tuning is leaky, and the abstract overclaims on temporal reasoning.","tokens_in":17715,"tokens_out":3685,"would_cite":true,"duration_ms":34967,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free frame selector that reads the question beats uniform sampling for video LLMs.","keywords":["video understanding","frame selection","Gumbel-Max trick","CLIP retrieval","multi-resolution adaptation","Video-LLM","long video QA","query-aware sampling"],"falsifier":"Run Q-Frame on a set of video questions whose answers are only determinable by the order or causality of events across timestamps, such as 'what happens immediately before X', and check whether the frames containing the evidence fall inside the top-K selection; the paper's appendix already shows a water-break timing question where both Q-Frame and uniform sampling fail, and a systematic drop on such a temporal-reasoning subtask would bound the method's scope.","tokens_in":16660,"feed_emoji":"🎞️","tokens_out":7428,"duration_ms":68096,"temperature":0.7,"pith_summary":"Q-Frame proposes that a video question can be answered better by choosing frames on the basis of the question itself, rather than taking frames at fixed intervals. The method uses a pretrained text-image matcher to score candidate frames against the query, adds Gumbel noise so the selection explores, and then assigns high, medium, or low resolution by relevance rank so more frames fit the same visual-token budget. The paper reports consistent gains over uniform sampling on MLVU, LongVideoBench, and Video-MME, with Qwen2-VL improving from 56.9 to 65.4 on MLVU, 53.5 to 58.4 on LongVideoBench, and 53.7 to 58.3 on Video-MME without subtitles. The significance, if the claim holds, is a training-free, model-agnostic upgrade that any Video-LLM can adopt.","feed_headline":"Query-aware frames push video LLMs past uniform sampling","feed_subtitle":"Training-free frame selection puts more useful frames in the token budget, adding up to 8.5 points for Qwen2-VL.","key_machinery":"Three components carry the argument. Cross-modal Query Retrieval (CQR) embeds candidate frames and the query with a CLIP-like model and computes inner-product similarity scores. Query-aware Frame Selection (QFS) applies a temperature-scaled softmax and the Gumbel-Max trick—adding independent Gumbel noise to the log-probabilities and taking the top K—which turns a deterministic retrieval score into a stochastic sampler without any training. Multi-Resolution Adaptation (MRA) then assigns each selected frame a resolution tier by rank, with the ratio $r(1)=4r(2)=16r(3)$, so token cost falls sharply for low-relevance frames; the resolution constraint $K + M/4 + N/16 = 8$ expresses the fixed budget in units of high-resolution frames. The Gumbel-Max trick is what makes the selection training-free, and the constraint is what lets more frames be presented under the same token limit.","core_discovery":"The paper's central claim is that the video frames an LLM needs for a question can be recovered from per-frame similarity to the question text, and that a fixed computational budget is best spent by giving the most relevant frames high resolution and less relevant frames low resolution, effectively multiplying how many frames that budget can carry. Q-Frame selects the top-K frames from a Gumbel-perturbed softmax over CLIP scores, then splits the selected frames into three resolution tiers satisfying $r(1)=4r(2)=16r(3)$; with Qwen2-VL the best configuration is 4 high-, 8 medium-, and 32 low-resolution frames, which the token budget treats as equivalent to 8 uniform frames. In the paper's evaluation this yields consistent improvements over uniform sampling across three long-video benchmarks and across open- and closed-source backbones, including GPT-4o. The authors explicitly do not claim the selection preserves temporal order; Q-Frame is positioned as query-aware keyframe retrieval plus resolution allocation.","pith_inferences":["Because selection is a per-frame retrieval step, the approach should transfer to other retrieval-style multimodal tasks such as video retrieval or grounded captioning, where the answer signal lives in a few frames.","The Gumbel noise is resampled for each query, so drawing several frame subsets and averaging the model's answers could act as an ensemble, a test the paper does not run.","The multi-resolution idea could be inverted: tasks that need spatial detail everywhere could trade total frame count for more high-resolution frames, converting the same token budget into coverage or detail as needed.","The method's own bad-case analysis predicts a clear boundary: temporal reasoning that requires ordering events is not handled, so any extension that preserves inter-frame order would directly widen the method's scope."],"forward_implications":["Video-LLM inference can be upgraded without fine-tuning, since any model that accepts visual tokens can ingest the selected frames and per-frame resolutions.","Longer videos become more tractable: sampling 128 candidate frames and passing 4 high-, 8 medium-, and 32 low-resolution frames keeps accuracy or improves it while using roughly the token budget of 8 uniform frames.","The token budget becomes a resolution-allocation problem, so fine-grained questions about small objects and OCR text benefit from reserving high resolution for the most relevant frames.","Closed-source API models also improve, because selection happens outside the model; the paper reports Q-Frame lifts GPT-4o on Video-MME without subtitles from 61.9 to 63.8."],"supporting_citations":[{"why":"Supplies the cross-modal similarity scores that drive retrieval; the whole selection depends on this pretrained text–image alignment.","marker":"[22]"},{"why":"Provides the Gumbel-Max trick that turns CLIP scores into a noisy categorical sample for frame selection without training.","marker":"[8]"},{"why":"Is the main open-source backbone used in experiments; its native dynamic resolution tokenization is what makes per-frame resolution assignment feasible.","marker":"[27]"},{"why":"Is the training-based comparison method whose protocol (128 candidate frames, 8 selected) Q-Frame adopts and seeks to beat.","marker":"[33]"},{"why":"Replaces CLIP for encoding longer queries in the retrieval-backbone ablation and achieves the best reported accuracy.","marker":"[37]"},{"why":"Provides the Video-MME benchmark used to evaluate the method with and without subtitles and across task categories.","marker":"[7]"},{"why":"Provides the LongVideoBench validation set used for main results and duration-bucketed analysis.","marker":"[29]"},{"why":"Provides the MLVU multi-task long-video benchmark where Q-Frame reaches its best headline number.","marker":"[41]"}],"fun_headline_variants":["Query-aware frame selection boosts video LLM accuracy","Training-free frame picker multiplies video token budget","Multi-resolution frames save tokens, improve video answers","Q-Frame: pick key frames per question, boost video LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a per-frame CLIP score against the question, plus random noise, is enough to pick the frames a video model needs, including questions whose answer depends on how events unfold over time rather than on what a single frame shows.","fun_headline_variants_meta":{"raw":{"variants":["Query-aware frame selection boosts video LLM accuracy","Training-free frame picker multiplies video token budget","Multi-resolution frames save tokens, improve video answers","Q-Frame: pick key frames per question, boost video LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1427,"prompt_tokens":933,"completion_tokens":494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":429}},"tokens_in":549,"tokens_out":494,"duration_ms":4940,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:10:41.520979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Q-Frame on a set of video questions whose answers are only determinable by the order or causality of events across timestamps, such as 'what happens immediately before X', and check whether the frames containing the evidence fall inside the top-K selection; the paper's appendix already shows a water-break timing question where both Q-Frame and uniform sampling fail, and a systematic drop on such a temporal-reasoning subtask would bound the method's scope.","supporting_citations":[{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-modal similarity scores that drive retrieval; the whole selection depends on this pretrained text–image alignment."},{"cited_title":"Frame-voyager: Learning to query frames for video large language models","cited_arxiv_id":null,"evidence_quote":"Is the training-based comparison method whose protocol (128 candidate frames, 8 selected) Q-Frame adopts and seeks to beat."},{"cited_title":"Longvideobench: A benchmark for long-context interleaved video-language understanding","cited_arxiv_id":null,"evidence_quote":"Provides the LongVideoBench validation set used for main results and duration-bucketed analysis."}],"review_version":1}