{"id":"bf4fa1d9-3f9c-4fed-a1e3-c3f04f9a4451","arxiv_id":"2501.19098","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"∞-Video extends ∞-former continuous-attention ideas to video Q-formers, adding a training-free long-term memory that improves long-video question answering in several benchmarks.","lead":"The paper adds a continuous-time long-term memory to video Q-former models so they can process long videos without being retrained. It reports gains on several video question-answering benchmarks, mainly for its Video-LLaMA-based variant with sticky memory sampling.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Similarity-based multiple-choice evaluation is the load-bearing weakness; pooling is secondary.","rationale":"The paper's central claim is that the continuous long-term memory in Eq. 16 improves long-video question answering, with the main evidence being the sticky-memory gains in Tables 1 and 3. The most load-bearing concern is the evaluation protocol: for multiple-choice benchmarks, the appendix describes a nonstandard similarity-based answer selection rather than the standard option-in-prompt protocol. Since the same protocol is used for the no-LTM baseline and the sticky variant, the relative ordering could still be meaningful, but the absence of variance estimates and the sensitivity of similarity matching to answer style make it impossible to rule out artifacts. The reader's weakest_assumption focused on average pooling of patch embeddings; that is a plausible limitation of the LTM representation, but it does not explain why sticky beats no-LTM under identical conditions. I therefore partially agree with the reader: their rationale mentions the evaluation issues, but their designated weakest assumption is different. A concrete re-evaluation under both protocols with confidence intervals would settle the concern. The code release is a positive factor, and the proposed test is feasible. This does not change the reader's CONDITIONAL verdict; it sharpens the condition under which the paper's claims could be accepted.","tokens_in":21662,"tokens_out":5128,"duration_ms":51308,"concrete_test":"Re-evaluate the no-LTM, uniform, and sticky variants of ∞-Video LLaMA on NeXT-QA and EgoSchema using the standard multiple-choice protocol (options included in the prompt, answer letter parsed) and also using the paper's similarity protocol with three different similarity measures. Compute bootstrap confidence intervals over test questions. If the sticky-versus-no-LTM margin does not persist with non-overlapping confidence intervals under both protocols, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A.1 describes how NeXT-QA and EgoSchema accuracies in Table 1 are obtained: the model is prompted with the question only, and the answer option is chosen by text similarity between the generated response and the options via LangChain. This differs from the standard protocol used to report prior baselines, so cross-method comparisons are uncontrolled. More importantly, the similarity step can systematically favor longer or more specific answers; sticky memories produce more detailed outputs, so the sticky-versus-no-LTM gains (+3.5 on NeXT-QA, +6.0 on EgoSchema) may reflect answer style rather than the LTM. No error bars or significance tests are reported, and the MovieChat open-ended results rely on a GPT-3.5 judge with the same missing variance. The average-pooling assumption in Section 3 is a genuine limitation but is not the load-bearing issue: it applies equally to all variants, and the LTM still improves over no-LTM, so the pooling concern does not invalidate the reported ordering. The evaluation protocol does.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ∞-Video, a training-free extension of video Q-former based models (Video-LLaMA and VideoChat2) that augments the existing short-term memory (STM) cross-attention over a frame chunk with a continuous-time long-term memory (LTM) built from previous chunks. The LTM is constructed by average-pooling frame patch embeddings, fitting a continuous signal with a basis-function ridge regression, and attending to that signal with a Gibbs density derived from the query-key similarity. A weighting factor α in Eq. (16) mixes the vanilla chunk-level context with the LTM context, and a 'sticky' sampling procedure allocates more LTM samples to regions of high prior attention density. The authors evaluate the approach on NExT-QA, EgoSchema, VideoMME, and MovieChat-1K, reporting gains for the sticky LTM over the no-LTM baseline for the Video-LLaMA-based models, and smaller or mixed gains for the VideoChat2-based models.","tokens_in":21981,"tokens_out":3043,"duration_ms":34618,"significance":"If the reported empirical gains are reliable, the paper offers a useful, plug-in way to extend short-video Q-former models to longer inputs without fine-tuning, and it does so with a clearly described mechanism that reuses the authors' own ∞-former machinery. The manuscript is generally well written, the architecture is specified in enough detail to be reimplemented, and the authors ship the code. The main significance is the demonstration that a continuous-time memory with attention-weighted sampling can beat simple uniform memories and no-memory baselines on long-video QA. However, the decisive evaluation evidence is currently weakened by a nonstandard multiple-choice protocol and by the absence of variance or significance information, so the central claim should be treated as promising but not yet established.","major_comments":[{"comment":"The multiple-choice accuracies on NExT-QA and EgoSchema are obtained by prompting the model with the question only and then choosing the answer option with the highest text similarity between the generated open-ended response and the options, computed via LangChain. This differs from the forced-choice scoring protocol used by the cited baselines (e.g., Video-LLaMA, MovieChat+, VideoTree), making the cross-model comparisons in Table 1 uncontrolled. More importantly, the sticky LTM tends to produce longer and more detailed answers, so the sticky-versus-no-LTM gains (+3.5 on NExT-QA and +6.0 on EgoSchema for ∞-Video LLaMA) could partly reflect answer style rather than improved video understanding. Please report results under the standard multiple-choice protocol, or at minimum run the same similarity-based protocol on all baselines and show that the relative ordering is preserved.","section":"Appendix A.1, Tables 1 and 3"},{"comment":"No variance or statistical significance is reported for any of the central comparisons. The MovieChat open-ended scores are produced by a GPT-3.5 judge, which is stochastic, and the authors themselves note in Appendix B.1 that results are affected by 'the inherent variability in ChatGPT's outputs,' yet no repeated evaluations, confidence intervals, or significance tests are provided. Since the headline claim is a 3-6 point accuracy gain, the reader cannot tell whether these differences are within run-to-run noise. Please report means and standard deviations over multiple runs (or seeds) for the LTM variants and the no-LTM baselines, and report a judge-agreement or significance analysis for the GPT-3.5-based metrics.","section":"Tables 1 and 3, Appendix B.1"},{"comment":"The LTM is built from frame embeddings obtained by average-pooling the P patch embeddings of each frame. This discards spatial information that the original video Q-former cross-attention may need, and the paper does not measure the information loss from this pooling step or ablate alternative pooling strategies. The concern is not fatal for the sticky-versus-no-LTM comparison, since all variants share the same pooling, but it does limit the claim that the method 'fully' processes video information and leaves open the possibility that the LTM gain would change under a higher-fidelity frame representation.","section":"Section 3, before Eq. (7)"},{"comment":"The linear combination Z = αZ_STM + (1 − α)Z_LTM is introduced as a post-hoc interpolation between two context vectors that are not obviously commensurable: Z_STM is computed from the original patch-level STM attention, while Z_LTM is computed from pooled frame-level continuous attention with a different key/value distribution. The paper does not validate that the pretrained Q-former layers remain well-behaved under this interpolation, apart from the empirical sweeps in Figure 5. A small diagnostic showing the sensitivity of downstream accuracy to α, or an analysis of the cosine similarity between Z_STM and Z_LTM, would strengthen the architectural justification.","section":"Section 3.4, Eq. (16)"}],"minor_comments":[{"comment":"The denominator in the Gibbs density is written as ∫ exp(s_i^h(t')) dt' without parentheses or a variable of integration in the displayed expression; please clarify the notation.","section":"Eq. (10)"},{"comment":"The entry 'Vicuna-7Ball/2048' appears to be a formatting error; it should read something like 'Vicuna-7B, all/2048 frames'.","section":"Table 1"},{"comment":"The column header 'A vg' is misspelled, and the baseline 'Video-LLaV A' contains an unintended space; also the 'Medium' and 'Long' averages for ST-LLM (36.8, 31.1) do not average to 37.9, so it should be clarified whether 'Avg' includes a short-video split.","section":"Table 2"},{"comment":"The GPT-3.5 prompt requests an integer score but gives '4.8' as the example output; this inconsistency should be fixed.","section":"Appendix A.3, Listing 1"},{"comment":"The Chandra et al. (2025) reference is cited with 'XX(XX):XX–XX' for volume and pages; this should be completed before publication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a clean adaptation of the authors' own ∞-former to video Q-formers, and the core idea is simple and reasonably presented. The main risk is that the empirical evaluation, as currently reported, does not yet support the strength of the claims: the multiple-choice protocol is nonstandard and the open-ended evaluation lacks variance reporting. If the authors can re-run the comparisons under the standard protocol and provide error bars, the paper could become a solid contribution. I would also gently note that the novelty relative to ∞-former and MovieChat is incremental, so the framing should emphasize the controlled comparisons and the new Gibbs-density continuous attention rather than the general idea of memory consolidation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a training-free extension of the authors' own ∞-former to video Q-formers, and the LTM mechanism is a direct adaptation. The real question is whether the reported gains are real, and right now the evaluation method prevents us from saying yes.\n\nThe new piece is the Gibbs-density continuous attention over pooled frame embeddings, replacing the Gaussian in ∞-former, plus the sticky-memory sampling that concentrates memory where attention has been high. That is a sensible and clean adaptation. They ship code, include ablations over α and N, and cite MovieChat/MovieChat+ and ∞-former properly. The architecture is coherent.\n\nThe soft spots are in the evaluation. For NeXT-QA and EgoSchema, the model is prompted with the question only, and the answer option is chosen by LangChain text similarity between the generated response and the options. That is not the protocol used by the baselines they compare against, so the cross-method numbers in Table 1 are not apples-to-apples. Worse, sticky memories produce longer, more detailed answers, and a similarity-based matcher can systematically prefer those—so the +3.5 and +6-point sticky-over-no-LTM gaps may reflect answer style rather than memory quality. No error bars or significance tests are reported anywhere, and the MovieChat open-ended results rely on a GPT-3.5 judge with the same missing variance.\n\nSecond, the gains are concentrated on the weaker Video-LLaMA base. On VideoChat2, the sticky variant is essentially flat (78.1 vs 78.1, 64.8 vs 64.6). The paper acknowledges this, but it means the practical benefit is base-dependent.\n\nOne more thing worth noting: their no-LTM baseline already uses all frames (2048), beating Video-LLaMA's 32-frame result by a wide margin. So part of the improvement over the original model comes from more frames, not from the LTM.\n\nThe average pooling of P patch embeddings into one embedding per frame is a genuine limitation, but it is secondary—it applies to all variants, and the LTM still changes the ordering. The evaluation protocol is the load-bearing issue.\n\nFor a reader working on long-video memory or training-free adaptation, this is a useful paper to know about. It deserves a serious referee, but the referee should ask for standard multiple-choice evaluation (or at least the same protocol as the baselines), error bars, and an analysis that separates answer length/style from memory content.\n\nRecommendation: send to review, conditional on fixing the evaluation.","headline":"A clean training-free memory extension for video Q-formers, but the reported gains rest on a non-standard similarity-based evaluation that needs fixing before the claims carry weight.","tokens_in":22408,"tokens_out":2604,"would_cite":true,"duration_ms":25693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims short-video language models can understand arbitrarily long videos without retraining, by blending chunk-local attention with a continuous-time memory, $Z = \\alpha Z_{\\mathrm{STM}} + (1-\\alpha)Z_{\\mathrm{LTM}}$.","keywords":["long-video understanding","video question answering","continuous attention","long-term memory","memory consolidation","training-free adaptation","video Q-former","sticky memories"],"falsifier":"Build a long-video question set whose answers require fine spatial detail inside a single frame (for example, the color or position of a small object); if the sticky-memory gain over the no-LTM baseline disappears or turns negative on those questions while staying positive on coarse temporal questions, the average-pooling step is the limiting factor.","tokens_in":21443,"feed_emoji":"🧠","tokens_out":17660,"duration_ms":136852,"temperature":0.7,"pith_summary":"∞-Video sets out to prove that video-language models trained only on short clips can understand arbitrarily long videos without any retraining, by giving the video Q-former — a module that compresses visual embeddings into a fixed set of video tokens — a continuous-time long-term memory alongside its usual per-chunk attention. The load-bearing move is to replace chunk-local attention with a weighted mixture of a local short-term memory and a compressed continuous memory, $Z = \\alpha Z_{\\mathrm{STM}} + (1-\\alpha)Z_{\\mathrm{LTM}}$, updated as chunks arrive. On the Video-LLaMA backbone the paper reports accuracy gains over the no-memory baseline of 41.1 vs 37.6 on NeXT-QA, 46.8 vs 40.8 on EgoSchema, and 72.2 vs 68.0 on MovieChat-1K, with relevance-guided 'sticky' sampling consistently better than uniform sampling. If correct, pretrained short-video architectures can scale to hour-long footage in a single pass at no training cost.","feed_headline":"Zero-training sticky memory wins six more points on video questions","feed_subtitle":"Relevance-weighted memory lets off-the-shelf video models handle hour-long footage without retraining.","key_machinery":"The load-bearing object is the continuous-time long-term memory inserted into the video Q-former's cross-attention. Frames are grouped into chunks, each frame's $P$ patch embeddings are average-pooled into one embedding, and the chunk is turned into a continuous signal $x(t) = B^\\top\\psi(t)$ by ridge regression onto $N$ basis functions (Eq. 5). Attention over this signal replaces discrete softmax with a Gibbs density $p_i^h(t) = \\exp(s_i^h(t)) / \\int \\exp(s_i^h(t'))\\,dt'$, so the context is the expected projected value (Eqs. 10–11). Memory consolidation contracts the old signal by a factor $\\tau$, samples it at $T$ locations, and re-regresses it together with the new chunk (Eqs. 12–14); 'sticky memories' choose those locations from the histogram of previous attention densities (Eq. 15), giving more memory resolution to moments the model already attends to. The output context is $Z = \\alpha Z_{\\mathrm{STM}} + (1-\\alpha)Z_{\\mathrm{LTM}}$ (Eq. 16), and chunk outputs are combined into the LLM input by a running average (Eq. 17).","core_discovery":"The central claim is that the cross-attention of a video Q-former can be augmented with a continuous-time long-term memory that consolidates information from previous frame chunks, and that this training-free change lets short-context video models process unbounded video in a single pass. The video is split into chunks; each chunk produces its usual local attention (the short-term memory, STM), while a smoothed continuous signal $x(t) = B^\\top\\psi(t)$, built by ridge regression over the chunk's frame embeddings, is attended to through a Gibbs probability density, forming the LTM. Each new chunk contracts the old signal into the interval $[0,\\tau]$ and re-regresses it together with the new chunk, so older content is gradually 'forgotten' while the most-attended regions keep higher resolution through relevance-based sampling. The final context is a weighted sum $Z = \\alpha Z_{\\mathrm{STM}} + (1-\\alpha)Z_{\\mathrm{LTM}}$, and per-chunk outputs are folded into a running-average token embedding that the LLM reads to answer the question. The paper argues that this dynamic consolidation — not merely seeing more frames — is what drives the reported gains, which appear mainly for the Video-LLaMA backbone and weakly or not at all for the stronger VideoChat2.","pith_inferences":["A direct test of the paper's weak spot would be to feed the LTM richer per-frame summaries than the average pool (for instance, pooled mean plus a handful of top patch tokens); improved accuracy would confirm the pooling step is the bottleneck the paper leaves unmeasured.","The sticky-memory histogram is effectively a training-free saliency map over video time, so one could check whether its peaks align with human-annotated event boundaries or key moments, connecting the mechanism to cognitive work on event segmentation.","Because the consolidation step is modality-agnostic, the same continuous-time LTM could wrap Q-former-style adapters for audio or sensor streams, where a chunk is a time window rather than a frame block.","The method is compositional: several memories at different timescales (multiple contraction factors $\\tau$) could be stacked to build a hierarchical video memory, or the LTM could be plugged underneath trained long-context models."],"forward_implications":["Any pretrained short-video model with a video Q-former can in principle be pointed at hour-long footage in a single pass, with fixed memory cost and no retraining.","Relevance-guided ('sticky') memory sampling beats uniform sampling for the Video-LLaMA family, so where memory is allocated matters more than raw frame count.","The best configuration keeps both local and global components ($\\alpha \\approx 0.9$); removing either the STM or the LTM lowers accuracy, identifying the weighted mixture itself as the operative mechanism.","For the already strong VideoChat2 backbone the gains are small or absent, which the paper attributes to the model being near saturation on in-domain benchmarks, implying the method's headroom depends on the base model's weakness."],"supporting_citations":[{"why":"Supplies the ∞-former continuous-attention infinite-memory machinery (τ contraction, sampling, ridge regression) that ∞-Video adapts to video Q-formers.","marker":"Martins et al., 2022b"},{"why":"Introduces continuous attention, replacing attention probability mass with a density; the foundation the new Gibbs-density mechanism extends.","marker":"Martins et al., 2020"},{"why":"Provides the multivariate ridge regression used to compute the basis-function coefficients B in Eq. 5, the backbone of the continuous signal construction.","marker":"Brown & Zidek, 1980"},{"why":"Video-LLaMA, the base short-video architecture whose temporal Q-former is augmented with the LTM and which provides the 7B Vicuna-based backbone.","marker":"Zhang et al., 2023b"},{"why":"VideoChat2, the second base architecture adapted, bringing a stronger Q-former, UMT-L encoder, and Mistral-7B LLM.","marker":"Li et al., 2024"},{"why":"MovieChat, the closest training-free long-video baseline and the source of the MovieChat-1K open-ended benchmark.","marker":"Song et al., 2023"},{"why":"MovieChat+, the question-aware sparse-memory baseline that ∞-Video must beat under the same Video-LLaMA base.","marker":"Song et al., 2024"},{"why":"NeXT-QA, the multiple-choice benchmark whose accuracy is the primary short-video comparison.","marker":"Xiao et al., 2021"},{"why":"EgoSchema, the medium-length egocentric benchmark where the sticky-memory gain is largest.","marker":"Mangalam et al., 2023"},{"why":"Video-MME, the hour-scale benchmark used to test very long videos for the VideoChat2 variants.","marker":"Fu et al., 2024"}],"fun_headline_variants":["Training-free sticky memory gives video QA a six-point boost","Off-the-shelf video models go long with continuous-time memory","Sticky memories let Video-LLaMA handle hour-long footage untrained","Continuous-time memory consolidates long video without retraining","Six-point gain on video QA from training-free memory consolidation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that average-pooling each frame's patch embeddings into a single vector still preserves enough spatial detail for the continuous memory to answer fine-grained questions, and the paper never measures how much information that pooling step discards.","fun_headline_variants_meta":{"raw":{"variants":["Training-free sticky memory gives video QA a six-point boost","Off-the-shelf video models go long with continuous-time memory","Sticky memories let Video-LLaMA handle hour-long footage untrained","Continuous-time memory consolidates long video without retraining","Six-point gain on video QA from training-free memory consolidation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1432,"prompt_tokens":935,"completion_tokens":497,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":551,"tokens_out":497,"duration_ms":4431,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:16:30.376176+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a long-video question set whose answers require fine spatial detail inside a single frame (for example, the color or position of a small object); if the sticky-memory gain over the no-LTM baseline disappears or turns negative on those questions while staying positive on coarse temporal questions, the average-pooling step is the limiting factor.","supporting_citations":[{"cited_title":"Sparse and continuous attention mechanisms","cited_arxiv_id":null,"evidence_quote":"Introduces continuous attention, replacing attention probability mass with a density; the foundation the new Gibbs-density mechanism extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multivariate ridge regression used to compute the basis-function coefficients B in Eq. 5, the backbone of the continuous signal construction."},{"cited_title":"Next-qa: Next phase of question-answering to explaining temporal actions","cited_arxiv_id":null,"evidence_quote":"NeXT-QA, the multiple-choice benchmark whose accuracy is the primary short-video comparison."}],"review_version":1}