{"id":"9f391442-4344-465c-a1cc-f4f88512a569","arxiv_id":"2412.09907","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A question-adaptive visual compressor condenses each frame into 64 context tokens, improving long-term video QA accuracy while using fewer memory tokens than prior memory-augmented methods.","lead":"The authors propose IQViC, a visual compressor that squeezes each video frame into a few question-conditioned tokens before feeding a language model, cutting memory use for long-video QA. On a new filtered subset of InfiniBench plus three short-video benchmarks, it reports higher accuracy than three prior memory-based video LLMs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 4.8-point long-term gain rests on an unvalidated self-filtered benchmark: name-based filtering leaves questions requiring dialogue/prior plot knowledge (e.g., Fig. 5b), so InfiniBench-Vision accuracy may not measure frame-only understanding.","rationale":"The reader's weakest-assumption analysis identifies benchmark validity as the key risk, and the manuscript text supports that concern with concrete counterexamples inside its own supplementary material. The filtering heuristic is explicitly name-based: it removes questions containing detected character names, but does not assess whether the remaining questions are visually answerable. Many surviving questions reference events, motivations, and background knowledge that would normally be conveyed through dialogue or subtitles, so the benchmark is not a clean video-frame-only test. If the benchmark is contaminated, the accuracy ranking on it does not establish the central claim of long-term understanding superiority. The absence of error bars and the use of only 599 questions make the 4.8-point difference additionally fragile, but the root issue is construct validity rather than statistical noise. I considered the question-conditioned encoding advantage as an alternative concern, since IQViC re-encodes the entire video for each question and therefore uses the question as a prior during compression; this is a genuine architectural feature and is acknowledged in the limitations, but it does not by itself invalidate the benchmark comparison as directly as the unvalidated benchmark does. The short-term results on NExT-QA, MSVD-QA, and MSRVTT-QA are independent evidence that the method has some merit, but they do not rescue the long-term SOTA claim. A conditional verdict requiring human validation of InfiniBench-Vision or evaluation on an external frame-only benchmark is therefore appropriate; no change to the reader's verdict is needed.","tokens_in":16962,"tokens_out":5759,"duration_ms":70134,"concrete_test":"Take 100 randomly sampled QA pairs from InfiniBench-Vision. For each, show three independent annotators the video frames used by the method (without audio or subtitles) and ask whether the ground-truth answer can be determined from visual content alone; compute majority labels and inter-annotator agreement. If more than roughly 10% are judged non-visually answerable, the benchmark is not a valid frame-only test, and the Table 2 comparison should be re-run on MLVU or LVBench (or a human-validated subset) before accepting the superiority claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is IQViC's superiority in long-term video understanding (Table 2: 49.1 vs 44.3, 640 vs 681 tokens). The load-bearing condition is that InfiniBench-Vision measures frame-only long-term understanding. That condition is not established. The curation pipeline (Supplementary §6) removes only QA pairs containing GPT-4o-mini-detected character names; it makes no check that the surviving 599 questions are answerable from frames. The supplementary's own examples refute this: 'What role does the connection to the previous murders in Florida play in solving the recent cases?' and 'How does the plastic surgeon's anticipated disappearance influence the urgency of the investigation?' require dialogue, plot facts, and off-screen narrative, not visual content. Thus the benchmark is likely contaminated with questions that no frame-only model can answer correctly, and the ranking on such a set does not demonstrate visual understanding. Moreover, with 599 questions and single runs, the 4.8-point gap has no error bars; a handful of GPT-judged answers could flip it. Since the same filtered set is used for all methods, this does not bias toward IQViC specifically, but it invalidates the claim that the comparison measures the proposed method's long-term visual understanding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IQViC, an in-context, question-adaptive visual compressor for long-term video understanding in large multimodal models (LMMs). The compressor is a transformer-based module that, conditioned on the text question, compresses visual embeddings of each frame into a small set of context tokens stored in a context memory with fixed capacity; temporal compression merges similar adjacent memory embeddings. Training is performed in two steps: first, the visual compressor and embedding lookup are trained on image QA datasets; second, the LLM decoder is fine-tuned on video QA data. The authors introduce InfiniBench-Vision, a filtered subset of InfiniBench, and report results on it and on short-term benchmarks (NExT-QA, MSVD-QA, MSRVTT-QA), claiming state-of-the-art long-term accuracy (49.1% vs. 44.3% for Flash-VStream) with fewer memory tokens (640 vs. 681). The central long-term claim depends on the validity of InfiniBench-Vision as a frame-only benchmark, which is not established.","tokens_in":17145,"tokens_out":5090,"duration_ms":54026,"significance":"The architectural idea is clear and potentially useful: conditioning the visual memory on the question is a natural way to reduce memory tokens while retaining task-relevant information. The paper gives a reproducible training recipe, detailed hyperparameters, and an ablation showing the compressor outperforms average pooling on LLaVA-Bench. The short-term results on standard benchmarks are informative. However, the headline contribution—superior long-term video understanding—is not currently supported because the evaluation rests on an unvalidated, self-constructed benchmark (InfiniBench-Vision) and a single-run comparison with no error bars. If the long-term result were established, the paper would be a solid contribution to efficient long-video LMMs; as it stands, the central claim is conditional on resolving the benchmark-validity and statistical-robustness concerns.","major_comments":[{"comment":"The central long-term evaluation uses InfiniBench-Vision, a dataset created by removing QA pairs whose questions contain character names detected by GPT-4o-mini. The curation process in Supplementary §6 does not verify that the remaining 599 questions are answerable from video frames alone. The manuscript's own Figure 5b shows that several surviving questions require dialogue or plot knowledge, e.g., 'What role does the connection to the previous murders in Florida play in solving the recent cases?' and 'How does the plastic surgeon's anticipated disappearance influence the urgency of the investigation?' These cannot be answered from visual frames alone. Consequently, Table 2's 49.1% accuracy and the claimed 4.8-point improvement over Flash-VStream may be an artifact of benchmark construction rather than evidence of superior frame-only long-term understanding. To support the claim, the authors need to provide human validation of visual answerability, compare performance on the original InfiniBench, and analyze how filtering changes question types and difficulty.","section":"Section 4.1, Supplementary §6"},{"comment":"The long-term comparison is based on a single run on 599 QA pairs, with no error bars, confidence intervals, or significance tests. A 4.8-percentage-point gap could plausibly be within noise; a small number of GPT-judged answers could flip the ranking. The same lack of statistical grounding applies to the short-term benchmarks in Table 3. The authors should report multiple runs, bootstrapped confidence intervals, or at least a sensitivity analysis of the GPT-based evaluation.","section":"Section 4.2, Table 2"},{"comment":"The long-term comparison includes only MovieChat, MA-LMM, and Flash-VStream. The paper cites other recent long-video LMMs (LongVA, MM-Vid, Artemis, LongVILA) in the related work but does not evaluate them. The claim of 'superiority over state-of-the-art methods' is therefore not established by the presented evidence. At minimum, the claim should be restricted to the compared methods, or additional baselines should be evaluated.","section":"Table 2 and Related Work §2.2"},{"comment":"The manuscript states in Section 3.6 that IQViC 'enables online QA through the parallelization of context memory construction and response generation,' but Section 7 (Limitation) admits that 'the current requirement to reprocess the entire video for each query is computationally expensive and impractical for online, interactive, long-video understanding applications.' Because the visual compressor is question-conditioned, every new question requires re-encoding and re-compressing the entire video, so the token-count advantage does not necessarily translate to lower total compute per question. The authors should quantify the total computational cost (e.g., FLOPs or latency) per query for IQViC versus baselines and reconcile the online-QA claim with the stated limitation.","section":"Section 3.6 vs. Section 7"}],"minor_comments":[{"comment":"The abstract refers to 'a new dataset based on InfiniBench' without noting that it is a filtered subset whose visual-answerability is not validated; please clarify this in both the abstract and the dataset description.","section":"Abstract and Section 4.1"},{"comment":"IQViC's NExT-QA accuracy (49.8) is slightly lower than MovieChat's (49.9); the text claims state-of-the-art performance 'according to the scores,' but the accuracy trade-off should be acknowledged explicitly.","section":"Section 4.2, Table 3"},{"comment":"The case-study question ('What message does the movie convey about the concept of fooling ourselves to get through life?') is itself arguably not answerable from visual frames alone, which weakens the demonstration; choosing an example whose frame-only answerability is clear would be more convincing.","section":"Figure 4"},{"comment":"The phrase 'inferent context window' appears to be a typo; it should be 'inference context window.'","section":"Section 1, first paragraph"},{"comment":"The statement that 'due to licensing restrictions, InfiniBench is the only long-term video understanding benchmark dataset accessible to us' is a significant caveat that should appear in the main text rather than only in the supplementary material.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central long-term claim is not supported by the current evidence because the self-constructed InfiniBench-Vision benchmark is not validated, and the authors' own supplementary examples show that the name-based filtering leaves questions requiring dialogue and plot knowledge. The method itself is well-specified and the short-term results are of interest, so the issues may be addressable with additional experiments, human validation of the benchmark, statistical robustness checks, and more cautious claims. If the benchmark cannot be validated, the long-term superiority claim should be substantially downgraded or removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: IQViC is a reasonable extension of in-context compression to video, and the short-term results and ablations give me some confidence the method works. But the headline claim—4.8 points over Flash-VStream on InfiniBench-Vision—sits on a benchmark the authors built themselves, and their own supplementary examples show it doesn't isolate frame-only understanding. Treat that number with skepticism.\n\nWhat's actually good: The core idea is simple: use the LLM itself to compress visual features into a small set of question-conditioned context tokens, then store them in a memory bank. It's a sensible adaptation of ICAE, and the two-step training (image QA first, then video QA) is practical. The ablation shows the compressor beats average pooling by a wide margin at the same compression ratio, which suggests the method is learning something real. The short-term VQA numbers are competitive, and on MSRVTT-QA they beat Flash-VStream by 4.3 points. That's credible.\n\nSoft spots: The long-term benchmark is the problem. The filtering removes QA pairs containing character names, but many surviving questions still require dialogue or off-screen plot knowledge. Look at their own Fig. 5b: \"What role does the connection to the previous murders in Florida play...\" That is not visual. So InfiniBench-Vision likely contains many questions that no frame-only model can answer correctly, and the ranking on such a set doesn't tell you who understands the video better. Also, 599 questions, single runs, no error bars—the 4.8-point gap could easily flip. The baseline comparison is thin: only MovieChat, MA-LMM, Flash-VStream. No LongVILA, Video-LLaMA-2, or other newer long-video models. Calling it state-of-the-art overreaches.\n\nTo their credit, the limitation section openly admits the video must be reprocessed per question, which is a real practical cost. That honesty is appreciated.\n\nWho this is for: people working on memory-augmented video LMMs will get useful ideas here, especially the compressor design and the two-step training recipe. But the empirical superiority claim should be re-evaluated on a properly validated benchmark, with error bars and stronger baselines, before anyone relies on it. The paper deserves a serious referee, not a desk reject, because the core idea is sound and the method is clearly described. I'd ask for major revision: validate the benchmark, release code/data, run significance tests, add stronger baselines.","headline":"The compressor idea is worth your time, but the headline long-video result sits on a self-filtered benchmark that likely does not isolate visual understanding, so treat the 4.8-point claim with caution.","tokens_in":17744,"tokens_out":2172,"would_cite":false,"duration_ms":23507,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a question-conditioned visual compressor, IQViC, that stores only task-relevant context and claims to beat prior long-video QA models with fewer memory tokens.","keywords":["long-term video understanding","visual compressor","question-conditioned compression","context memory","video question answering","memory tokens","large multimodal models","InfiniBench-Vision"],"falsifier":"Have human annotators judge each of the 599 InfiniBench-Vision questions as answerable or not from video frames alone, then rerun MovieChat, MA-LMM, Flash-VStream, and IQViC only on the questions marked answerable; if the accuracy gap over Flash-VStream shrinks below four points, the claimed advantage is an artifact of the filtering.","tokens_in":16711,"feed_emoji":"🎬","tokens_out":8711,"duration_ms":82559,"temperature":0.7,"pith_summary":"The paper tackles long-term video question answering, where full frame sequences overflow the language model's context window and memory banks blur details. It proposes IQViC, a visual compressor that takes each video frame together with the text question and outputs a small set of context tokens, so the bounded context memory stores only question-relevant information. On a newly filtered 599-question subset of InfiniBench called InfiniBench-Vision, containing 265 videos averaging 49 minutes, the authors report 49.1% accuracy and a 2.6 score using 640 memory tokens, against Flash-VStream's 44.3% and 2.4 at 681 tokens. They also report top scores on NExT-QA, MSVD-QA, and MSRVTT-QA, and an ablation showing the compressor beats average pooling by more than ten accuracy points at the same token budget. The intended payoff is that selective, question-conditioned compression rather than larger memory banks is the efficient route to long-video understanding.","feed_headline":"Question-aware compression beats full-video memory in long-video QA","feed_subtitle":"640 memory tokens, 49.1% accuracy on InfiniBench-Vision; previous best used 681 tokens for 44.3%.","key_machinery":"The IQViC visual compressor is a transformer-based encoder, adapted from the in-context autoencoder idea, that concatenates the text prompt embedding, the frame's visual embedding, and a learnable context embedding lookup; the output states of the context tokens after self-attention are the compressed context. The context memory stores these C=64-token embeddings in L=10 slots, and when the memory is full a temporal compressor removes redundancy by averaging the pair of adjacent embeddings with the highest cosine similarity. The load-bearing mechanism is that attention across question, visual, and context tokens lets the context tokens carry only the visual content the question needs, so no pooling or downsampling of the full frame is required.","core_discovery":"The central claim is that conditioning visual compression on the question preserves the information needed to answer that question while discarding what is irrelevant, and that this is enough to outperform memory-augmented models that try to retain the whole video. The model encodes each frame with a frozen CLIP ViT-L/14 encoder, projects it, concatenates the text prompt and C=64 learnable context tokens, and reads the context-token outputs from a LoRA-tuned transformer compressor as the frame's compressed memory. A parameter-free temporal compressor merges the most similar adjacent memory entries when the L=10 slots fill. With this design, IQViC reaches 49.1% accuracy and a 2.6 score on InfiniBench-Vision using 640 total memory tokens, a 4.8-point accuracy gain over Flash-VStream at 681 tokens, and it achieves scores of 4.0 on NExT-QA, 4.0 on MSVD-QA, and 3.5 on MSRVTT-QA. The ablation on LLaVA-Bench in the wild shows 55.4% accuracy at C=64, more than ten points above average pooling at the same 11% compression ratio.","pith_inferences":["Because InfiniBench-Vision was built by deleting questions whose character names a language model judged non-visual, the reported 4.8-point lead is a claim about the filtered subset; re-running the same comparison on the unfiltered InfiniBench questions would show how much of the advantage depends on that filtering.","The authors leave implicit that the compressor is re-run on the whole video for every new question; a dynamic memory update that only re-encodes unseen frames would reduce per-question cost and is a natural architectural extension.","Extending IQViC to audio and subtitles, which the paper names as future work, would target exactly the questions removed by the name filter and could broaden the method to full multimodal long-video understanding."],"forward_implications":["Long-video QA can run with a fixed 640-token context memory instead of storing thousands of frame tokens, making inference and memory usage predictable.","The same compressor generalizes to short videos: IQViC reports the best score on NExT-QA, MSVD-QA, and MSRVTT-QA among compared methods.","Context-token count is a tunable cost-accuracy knob, with C=1 already giving 49.9% on LLaVA-Bench in the wild and C=64 giving 55.4%.","Two-stage LoRA training, image-QA for the compressor and video-QA for the decoder, avoids end-to-end video backpropagation while preserving accuracy.","Online QA is possible because context-memory construction can be parallelized with response generation, as in Flash-VStream."],"supporting_citations":[{"why":"Supplies the in-context autoencoder architecture that the visual compressor adapts from text to vision.","marker":"[46]"},{"why":"Provides the previous best long-video baseline, Flash-VStream, and the Video-ChatGPT-filtered training data used in Step 2.","marker":"[23]"},{"why":"Source benchmark InfiniBench, from which InfiniBench-Vision is curated.","marker":"[55]"},{"why":"Provides the LLaVA-v1.5 base model and MLP projector used for the visual compressor and decoder.","marker":"[29]"},{"why":"Baseline with short-term and long-term memory whose temporal merging by similarity is adapted for the context memory.","marker":"[21]"},{"why":"Baseline memory-augmented LMM whose memory bank and temporal compression are compared and extended.","marker":"[22]"},{"why":"Defines the MLVU annotation policy of removing content-specific character-name questions, adopted for InfiniBench-Vision.","marker":"[56]"},{"why":"LoRA is the parameter-efficient training method used in both training steps.","marker":"[47]"},{"why":"Supplies LLaVA-Bench in the wild, the image QA benchmark used in the compressor ablation.","marker":"[30]"}],"fun_headline_variants":["Question-adaptive video compressor beats full-video memory in long-video QA","Forget irrelevant frames: IQViC compresses videos by question for better QA","640 tokens, 49.1% accuracy: question-aware compression beats full-video memory","Long-video QA: compress by question, not by clip, to answer better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that deleting questions flagged as containing non-identifiable character names by an automated name detector leaves a benchmark that is genuinely answerable from video frames alone, and no human validation is provided for that filtering.","fun_headline_variants_meta":{"raw":{"variants":["Question-adaptive video compressor beats full-video memory in long-video QA","Forget irrelevant frames: IQViC compresses videos by question for better QA","640 tokens, 49.1% accuracy: question-aware compression beats full-video memory","Long-video QA: compress by question, not by clip, to answer better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1699,"prompt_tokens":1018,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":634,"tokens_out":681,"duration_ms":7263,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:35:08.934929+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators judge each of the 599 InfiniBench-Vision questions as answerable or not from video frames alone, then rerun MovieChat, MA-LMM, Flash-VStream, and IQViC only on the questions marked answerable; if the accuracy gap over Flash-VStream shrinks below four points, the claimed advantage is an artifact of the filtering.","supporting_citations":[{"cited_title":"In-context autoencoder for context compression in a large language model","cited_arxiv_id":null,"evidence_quote":"Supplies the in-context autoencoder architecture that the visual compressor adapts from text to vision."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-v1.5 base model and MLP projector used for the visual compressor and decoder."},{"cited_title":"MovieChat: From dense token to sparse memory for long video understanding","cited_arxiv_id":null,"evidence_quote":"Baseline with short-term and long-term memory whose temporal merging by similarity is adapted for the context memory."},{"cited_title":"MA-LMM: Memory-augmented large multimodal model for long-term video understanding","cited_arxiv_id":null,"evidence_quote":"Baseline memory-augmented LMM whose memory bank and temporal compression are compared and extended."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies LLaVA-Bench in the wild, the image QA benchmark used in the compressor ablation."}],"review_version":1}