Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

$\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims short-video language models can understand arbitrarily long videos without retraining, by blending chunk-local attention with a continuous-time memory, $Z = \alpha Z_{\mathrm{STM}} + (1-\alpha)Z_{\mathrm{LTM}}$.

desk verdict A clean training-free memory extension for video Q-formers, but the reported gains rest on a non-standard similarity-based evaluation that needs fixing before the claims carry weight. read the letter →

arxiv 2501.19098 v2 pith:5VZQP7GA submitted 2025-01-31 cs.CV cs.LG

Saul Santos , António Farinhas , Daniel C. McNamee , André F. T. Martins This is my paper · ORCID
classification cs.CVcs.LG
keywords long-videounderstandingvideoquestionansweringcontinuousattentionlong-termmemoryconsolidationtraining-freeadaptationQ-formerstickymemories
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

∞-Video sets out to prove that video-language models trained only on short clips can understand arbitrarily long videos without any retraining, by giving the video Q-former — a module that compresses visual embeddings into a fixed set of video tokens — a continuous-time long-term memory alongside its usual per-chunk attention. The load-bearing move is to replace chunk-local attention with a weighted mixture of a local short-term memory and a compressed continuous memory, $Z = \alpha Z_{\mathrm{STM}} + (1-\alpha)Z_{\mathrm{LTM}}$, updated as chunks arrive. On the Video-LLaMA backbone the paper reports accuracy gains over the no-memory baseline of 41.1 vs 37.6 on NeXT-QA, 46.8 vs 40.8 on EgoSchema, and 72.2 vs 68.0 on MovieChat-1K, with relevance-guided 'sticky' sampling consistently better than uniform sampling. If correct, pretrained short-video architectures can scale to hour-long footage in a single pass at no training cost.

What carries the argument

The load-bearing object is the continuous-time long-term memory inserted into the video Q-former's cross-attention. Frames are grouped into chunks, each frame's $P$ patch embeddings are average-pooled into one embedding, and the chunk is turned into a continuous signal $x(t) = B^\top\psi(t)$ by ridge regression onto $N$ basis functions (Eq. 5). Attention over this signal replaces discrete softmax with a Gibbs density $p_i^h(t) = \exp(s_i^h(t)) / \int \exp(s_i^h(t'))\,dt'$, so the context is the expected projected value (Eqs. 10–11). Memory consolidation contracts the old signal by a factor $\tau$, samples it at $T$ locations, and re-regresses it together with the new chunk (Eqs. 12–14); 'sticky memories' choose those locations from the histogram of previous attention densities (Eq. 15), giving more memory resolution to moments the model already attends to. The output context is $Z = \alpha Z_{\mathrm{STM}} + (1-\alpha)Z_{\mathrm{LTM}}$ (Eq. 16), and chunk outputs are combined into the LLM input by a running average (Eq. 17).

What would settle it

Build a long-video question set whose answers require fine spatial detail inside a single frame (for example, the color or position of a small object); if the sticky-memory gain over the no-LTM baseline disappears or turns negative on those questions while staying positive on coarse temporal questions, the average-pooling step is the limiting factor.

Watch

Extended reading notes

Core claim

The central claim is that the cross-attention of a video Q-former can be augmented with a continuous-time long-term memory that consolidates information from previous frame chunks, and that this training-free change lets short-context video models process unbounded video in a single pass. The video is split into chunks; each chunk produces its usual local attention (the short-term memory, STM), while a smoothed continuous signal $x(t) = B^\top\psi(t)$, built by ridge regression over the chunk's frame embeddings, is attended to through a Gibbs probability density, forming the LTM. Each new chunk contracts the old signal into the interval $[0,\tau]$ and re-regresses it together with the new chunk, so older content is gradually 'forgotten' while the most-attended regions keep higher resolution through relevance-based sampling. The final context is a weighted sum $Z = \alpha Z_{\mathrm{STM}} + (1-\alpha)Z_{\mathrm{LTM}}$, and per-chunk outputs are folded into a running-average token embedding that the LLM reads to answer the question. The paper argues that this dynamic consolidation — not merely seeing more frames — is what drives the reported gains, which appear mainly for the Video-LLaMA backbone and weakly or not at all for the stronger VideoChat2.

Load-bearing premise

The method assumes that average-pooling each frame's patch embeddings into a single vector still preserves enough spatial detail for the continuous memory to answer fine-grained questions, and the paper never measures how much information that pooling step discards.

Editorial extensions

If this is right

  • Any pretrained short-video model with a video Q-former can in principle be pointed at hour-long footage in a single pass, with fixed memory cost and no retraining.
  • Relevance-guided ('sticky') memory sampling beats uniform sampling for the Video-LLaMA family, so where memory is allocated matters more than raw frame count.
  • The best configuration keeps both local and global components ($\alpha \approx 0.9$); removing either the STM or the LTM lowers accuracy, identifying the weighted mixture itself as the operative mechanism.
  • For the already strong VideoChat2 backbone the gains are small or absent, which the paper attributes to the model being near saturation on in-domain benchmarks, implying the method's headroom depends on the base model's weakness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's weak spot would be to feed the LTM richer per-frame summaries than the average pool (for instance, pooled mean plus a handful of top patch tokens); improved accuracy would confirm the pooling step is the bottleneck the paper leaves unmeasured.
  • The sticky-memory histogram is effectively a training-free saliency map over video time, so one could check whether its peaks align with human-annotated event boundaries or key moments, connecting the mechanism to cognitive work on event segmentation.
  • Because the consolidation step is modality-agnostic, the same continuous-time LTM could wrap Q-former-style adapters for audio or sensor streams, where a chunk is a time window rather than a frame block.
  • The method is compositional: several memories at different timescales (multiple contraction factors $\tau$) could be stacked to build a hierarchical video memory, or the LTM could be plugged underneath trained long-context models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ∞-Video, a training-free extension of video Q-former based models (Video-LLaMA and VideoChat2) that augments the existing short-term memory (STM) cross-attention over a frame chunk with a continuous-time long-term memory (LTM) built from previous chunks. The LTM is constructed by average-pooling frame patch embeddings, fitting a continuous signal with a basis-function ridge regression, and attending to that signal with a Gibbs density derived from the query-key similarity. A weighting factor α in Eq. (16) mixes the vanilla chunk-level context with the LTM context, and a 'sticky' sampling procedure allocates more LTM samples to regions of high prior attention density. The authors evaluate the approach on NExT-QA, EgoSchema, VideoMME, and MovieChat-1K, reporting gains for the sticky LTM over the no-LTM baseline for the Video-LLaMA-based models, and smaller or mixed gains for the VideoChat2-based models.

Significance. If the reported empirical gains are reliable, the paper offers a useful, plug-in way to extend short-video Q-former models to longer inputs without fine-tuning, and it does so with a clearly described mechanism that reuses the authors' own ∞-former machinery. The manuscript is generally well written, the architecture is specified in enough detail to be reimplemented, and the authors ship the code. The main significance is the demonstration that a continuous-time memory with attention-weighted sampling can beat simple uniform memories and no-memory baselines on long-video QA. However, the decisive evaluation evidence is currently weakened by a nonstandard multiple-choice protocol and by the absence of variance or significance information, so the central claim should be treated as promising but not yet established.

major comments (4)
  1. [Appendix A.1, Tables 1 and 3] The multiple-choice accuracies on NExT-QA and EgoSchema are obtained by prompting the model with the question only and then choosing the answer option with the highest text similarity between the generated open-ended response and the options, computed via LangChain. This differs from the forced-choice scoring protocol used by the cited baselines (e.g., Video-LLaMA, MovieChat+, VideoTree), making the cross-model comparisons in Table 1 uncontrolled. More importantly, the sticky LTM tends to produce longer and more detailed answers, so the sticky-versus-no-LTM gains (+3.5 on NExT-QA and +6.0 on EgoSchema for ∞-Video LLaMA) could partly reflect answer style rather than improved video understanding. Please report results under the standard multiple-choice protocol, or at minimum run the same similarity-based protocol on all baselines and show that the relative ordering is preserved.
  2. [Tables 1 and 3, Appendix B.1] No variance or statistical significance is reported for any of the central comparisons. The MovieChat open-ended scores are produced by a GPT-3.5 judge, which is stochastic, and the authors themselves note in Appendix B.1 that results are affected by 'the inherent variability in ChatGPT's outputs,' yet no repeated evaluations, confidence intervals, or significance tests are provided. Since the headline claim is a 3-6 point accuracy gain, the reader cannot tell whether these differences are within run-to-run noise. Please report means and standard deviations over multiple runs (or seeds) for the LTM variants and the no-LTM baselines, and report a judge-agreement or significance analysis for the GPT-3.5-based metrics.
  3. [Section 3, before Eq. (7)] The LTM is built from frame embeddings obtained by average-pooling the P patch embeddings of each frame. This discards spatial information that the original video Q-former cross-attention may need, and the paper does not measure the information loss from this pooling step or ablate alternative pooling strategies. The concern is not fatal for the sticky-versus-no-LTM comparison, since all variants share the same pooling, but it does limit the claim that the method 'fully' processes video information and leaves open the possibility that the LTM gain would change under a higher-fidelity frame representation.
  4. [Section 3.4, Eq. (16)] The linear combination Z = αZ_STM + (1 − α)Z_LTM is introduced as a post-hoc interpolation between two context vectors that are not obviously commensurable: Z_STM is computed from the original patch-level STM attention, while Z_LTM is computed from pooled frame-level continuous attention with a different key/value distribution. The paper does not validate that the pretrained Q-former layers remain well-behaved under this interpolation, apart from the empirical sweeps in Figure 5. A small diagnostic showing the sensitivity of downstream accuracy to α, or an analysis of the cosine similarity between Z_STM and Z_LTM, would strengthen the architectural justification.
minor comments (5)
  1. [Eq. (10)] The denominator in the Gibbs density is written as ∫ exp(s_i^h(t')) dt' without parentheses or a variable of integration in the displayed expression; please clarify the notation.
  2. [Table 1] The entry 'Vicuna-7Ball/2048' appears to be a formatting error; it should read something like 'Vicuna-7B, all/2048 frames'.
  3. [Table 2] The column header 'A vg' is misspelled, and the baseline 'Video-LLaV A' contains an unintended space; also the 'Medium' and 'Long' averages for ST-LLM (36.8, 31.1) do not average to 37.9, so it should be clarified whether 'Avg' includes a short-video split.
  4. [Appendix A.3, Listing 1] The GPT-3.5 prompt requests an integer score but gives '4.8' as the example output; this inconsistency should be fixed.
  5. [References] The Chandra et al. (2025) reference is cited with 'XX(XX):XX–XX' for volume and pages; this should be completed before publication.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the LTM mechanism is self-contained and evaluated on external benchmarks; self-citations are foundational but not load-bearing.

full rationale

The paper's central claim, that a training-free continuous-time long-term memory improves long-video QA, is not equivalent to any fitted input or to a self-citation. The LTM in Eq. 11 and the consolidation update in Eqs. 12-14 are explicit algorithmic operations applied to chunked video embeddings, and the reported gains are measured on external benchmarks (NeXT-QA, EgoSchema, VideoMME, MovieChat) using the same no-LTM variant as the controlled baseline. The Gibbs density in Eq. 10 and the sticky sampling in Eq. 15 are defined within the paper rather than imported as an unexamined theorem, so there is no reduction of the 'prediction' to its own definition. The paper acknowledges that the continuous-attention machinery adapts the authors' earlier ∞-former work, but that prior work is used as a component source, not as the sole justification for the empirical outcome; the current paper's evaluation is independent of that citation. Appendix A.1 discloses the LangChain similarity-based multiple-choice scoring, which is a comparison-validity concern rather than a circularity: it does not make the LTM-vs-no-LTM comparison true by construction, though it may confound answer style with memory quality. No fitted parameter is renamed as a prediction, and no uniqueness theorem from the authors' prior work is invoked to forbid alternatives. The average-pooling limitation noted in Section 3 is a genuine information-loss risk, but it applies symmetrically to all variants and does not make the reported ordering definitional. Overall, the derivation chain is self-contained and externally benchmarked, so circularity is minimal.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities: the LTM is a compressed continuous-time signal. However, it relies on several free parameters (α, τ, N, T), the adequacy of ridge regression and average pooling, and the reliability of GPT-3.5 as an evaluator.

free parameters (4)
  • alpha (LTM/STM weighting) = 0.9 for main results; ablated over {0, 0.25, 0.5, 0.75, 0.95, 1}
    Chosen by hand or tuned per experiment, and directly controls the balance between STM and LTM (Eq. 16).
  • tau (forgetting factor) = 0.75 for most experiments, 0.5 for VideoMME
    Controls how much of the time interval is allocated to past memory in the contraction step (Section 3.2).
  • N (number of basis functions) = 256 for VideoChat2 variants, 1024 for Video-LLaMA variants (except NeXT-QA with 256)
    Controls memory granularity and is ablated; higher N generally helps.
  • T (number of sampled locations) = 1000 trapezoidal integration points; T for memory sampling not precisely specified
    The number of sample points used to evaluate the continuous signal is a resolution parameter that affects the trade-off between memory and computation.
assumptions (5)
  • domain assumption Ridge regression in Eq. 5 yields an adequate continuous representation of the pooled frame embeddings.
    Section 2.2 and 3.2 rely on this without evaluating reconstruction error or its effect on downstream QA.
  • domain assumption Average pooling over the P spatial embeddings preserves enough information for memory.
    Section 3 states that 'we perform average pooling over the P embeddings', central to the LTM construction.
  • domain assumption The video Q-former is frozen and its cross-attention can be mixed with continuous-attention outputs without distribution shift.
    The training-free claim (Section 1, 'training-free manner') depends on the frozen pretrained model accepting the modified context vectors.
  • domain assumption GPT-3.5 evaluation scores are a reliable measure of answer quality without human validation or inter-annotator agreement.
    Appendix A.3 uses GPT-3.5 as judge for MovieChat metrics; the paper cites variability in GPT outputs but does not report statistical significance.
  • ad hoc to paper The vanilla Q-former's cross-attention over the current chunk can be combined linearly with LTM attention via Eq. 16.
    The weighted sum is introduced directly in Section 3.4; no derivation or empirical justification is given for the linear combination form.

how reviews work

0 comments
Cite this review

Pith. "Pith review of $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation." pith.science (2026). https://pith.science/paper/5VZQP7GA

@misc{pith2026250119098,
  author       = {Pith},
  title        = {Pith review of: $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VZQP7GA}},
  note         = {Machine review of arXiv:2501.19098}
}
abstract

Current video-language models struggle with long-video understanding due to limited context lengths and reliance on sparse frame subsampling, often leading to information loss. This paper introduces $\infty$-Video, which can process arbitrarily long videos through a continuous-time long-term memory (LTM) consolidation mechanism. Our framework augments video Q-formers by allowing them to process unbounded video contexts efficiently and without requiring additional training. Through continuous attention, our approach dynamically allocates higher granularity to the most relevant video segments, forming "sticky" memories that evolve over time. Experiments with Video-LLaMA and VideoChat2 demonstrate improved performance in video question-answering tasks, showcasing the potential of continuous-time LTM mechanisms to enable scalable and training-free comprehension of long videos.

Figures

Figures reproduced from arXiv: 2501.19098 by the authors.

Figure 1
Figure 1. (Left) Overview of ∞-VIDEO (our approach) using Video LLaMA (Zhang et al., 2023b, gray arrows), which uses an additional spatial Q-former module, and VideoChat2 (Li et al., 2024, black arrows). We split the video into frame chunks and apply these models to each chunk. The Video Q-former module combines a weighted average of the STM, which is the attention for an individual chunk, with a continuous LTM that takes int… view at source ↗
Figure 2
Figure 2. Proposed Memory Consolidation Mechanism. and build the matrix Xpast = [x1, x2, ..., xT ] ⊤ ∈ R T ×e . Following this, we concatenate the current step context Xnew ∈ RM×d with the previous context Xpast ∈ R T ×e , resulting in the combined sequence: X = [Xpast, Xnew] ⊤ ∈ R (T +M)×e . (14) With the new and previous chunk contexts, we compute B, as described in Eq. 5, where the continuous signal is approximated using a… view at source ↗
Figure 3
Figure 3. (Top) LTM attention density on the [0, τ ] interval for the Interstellar trailer, using sticky memories in the final chunk of the ∞-Video LLaMA video Q-former’s last layer. (Bottom) The same attention density map, extended over the full t interval. 12 28 128 148 150 191 311 339 400 465 478 486 528 544 545 553 555 571 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Highest continuous attention density frames selected using sticky memories in the Interstellar trailer for ∞-Video LLaMA across 3 chunks. (Left) Interval: [0, τ 2 ]. (Middle) Interval: (τ 2 , τ ]. (Right) Interval: (τ, 1]. et al., 2024; Chen et al., 2024b), but few ada…
Figure 5
Figure 5. Figure 5: Ablation studies on the MovieChat dataset: Evaluation of accuracy and score metrics for various values of the number of basis functions N and the contribution of long-term memory α. In this section, we conduct ablation studies on ∞-Video LLaMA on the MovieChat dataset.…
Figure 6
Figure 6. Figure 6: Highest continuous attention density frames selected using uniform memories in the Interstellar trailer for ∞-Video LLaMA across 3 chunks. (Left) Interval: [0, τ 2 ]. (Middle) Interval: (τ 2 , τ ]. (Right) Interval: (τ, 1]. B.2. Qualitative Analysis In [PITH_FULL_IMAG…
Figure 7
Figure 7. Figure 7: Examples of ∞-Video LLaMA answers with uniform sampling and sticky memories for short and ultra-long videos. Italicized corresponds to the correct answer while underlined corresponds to the wrong answer or hallucination. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding

    cs.CV 2025-06 conditional novelty 7.0 of 10

    MF2 evaluates long-movie understanding by asking models to classify fact/fib claim pairs; the best model trails humans by 23.5 points in pairwise accuracy.

  2. See What You Need: Query-Aware Visual Intelligence through Reasoning-Perception Loops

    cs.CV 2025-08 reject novelty 5.0 of 10

    CAVIA, a training-free loop where an LLM directs a VLM to inspect specific video frames and repeats until confident, reports accuracy gains on EgoSchema, NExT-QA, and IntentQA, but those gains are not shown to come fr...

  3. Modern Hopfield Networks with Continuous-Time Memories

    cs.LG 2025-02 conditional novelty 5.0 of 10

    A Hopfield energy over a continuous-time memory function, updated by a Gibbs density, retrieves video frames and embeddings as well as discrete modern Hopfield networks while using fewer basis functions.

  4. Infinite Video Understanding

    cs.CV 2025-07 conditional novelty 3.0 of 10

    The paper argues that video understanding research should aim at processing streams of arbitrary, unbounded duration and outlines the challenges, directions, and metrics needed.

Reference graph

Works this paper leans on

62 extracted references · 40 canonical work pages · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Prompt design matters for computational social science tasks but in unpredictable ways, 2024

    Atreja, S., Ashkinaze, J., Li, L., Mendelsohn, J., and Hemphill, L. Prompt design matters for computational social science tasks but in unpredictable ways, 2024. URL https://arxiv.org/abs/2406.11980

    arXiv 2024

  3. [3]

    Neural machine translation by jointly learning to align and translate

    Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proc. of International Conference on Learning Representations, 2015

    work page 2015

  4. [4]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  5. [5]

    Balazevic, I., Shi, Y., Papalampidi, P., Chaabouni, R., Koppula, S., and Henaff, O. J. Memory consolidation enables long-context video understanding. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings ...

    work page 2024

  6. [6]

    F., Konkle, T., Alvarez, G

    Brady, T. F., Konkle, T., Alvarez, G. A., and Oliva, A. Visual long-term memory has a massive storage capacity for object details. Proceedings of the National Academy of Sciences, 105 0 (38): 0 14325--14329, 2008. doi:10.1073/pnas.0803390105

    work page doi:10.1073/pnas.0803390105 2008

  7. [7]

    Brown, P. J. and Zidek, J. V. Adaptive multivariate ridge regression. The Annuals of Statistics, 1980

    work page 1980

  8. [8]

    Empowering large language model for continual video question answering with collaborative prompting, 2024

    Cai, C., Wang, Z., Gao, J., Liu, W., Lu, Y., Zhang, R., and Yap, K.-H. Empowering large language model for continual video question answering with collaborative prompting, 2024. arXiv:2410.00771v2

    arXiv 2024

Show all 62 references
  1. [9]

    F., Jadhav, S

    Carr, M. F., Jadhav, S. P., and Frank, L. M. Hippocampal replay in the awake state: A potential substrate for memory consolidation and retrieval. Nature Neuroscience, 14: 0 147--153, 2011

    2011

  2. [10]

    Episodic and associative memory from spatial scaffolds in the hippocampus

    Chandra, S., Sharma, S., Chaudhuri, R., et al. Episodic and associative memory from spatial scaffolds in the hippocampus. Nature, XX 0 (XX): 0 XX--XX, 2025. doi:10.1038/s41586-024-08392-y

    2025 doi

  3. [11]

    Sharegpt4video: Improving video understanding and generation with better captions

    Chen, L., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Lin, B., Tang, Z., et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325, 2024 a

    2024 arXiv

  4. [12]

    Longvila: Scaling long-context visual language models for long videos, 2024 b

    Chen, Y., Xue, F., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., He, E., Yin, H., Molchanov, P., Kautz, J., Fan, L., Zhu, Y., Lu, Y., and Han, S. Longvila: Scaling long-context visual language models for long videos, 2024 b

    2024

  5. [13]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms

    Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., and Bing, L. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. URL https://arxiv.org/abs/2406.07476

    2024 arXiv

  6. [14]

    E., and et al

    Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., and et al. Vicuna: An open-source chatbot impressing gpt-4 with 90\ https://vicuna.lmsys.org, 2023

    2023

  7. [15]

    T., Schapiro, A

    Cowan, E. T., Schapiro, A. C., Dunsmoor, J. E., and Murty, V. P. Memory consolidation as an adaptive process. Psychonomic Bulletin & Review, 28: 0 1796--1810, 2021. doi:10.3758/s13423-021-01978-x. URL https://doi.org/10.3758/s13423-021-01978-x

    2021 doi

  8. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on...

    2021

  9. [17]

    The consolidation and transformation of memory

    Dudai, Y., Karni, A., and Born, J. The consolidation and transformation of memory. Neuron, 88: 0 20--32, 2015

    2015

  10. [18]

    Eva: Exploring the limits of masked visual representation learning at scale

    Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. 2022

    2022

  11. [19]

    Farinhas, A., Martins, A. F. T., and Aguiar, P. M. Q. Multimodal continuous visual attention mechanisms. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp.\ 1047--1056, October 2021

    2021

  12. [20]

    Frankland, P. W. and Bontempi, B. The organization of recent and remote memories. Nature Reviews Neuroscience, 6 0 (2): 0 119--130, 2005

    2005

  13. [21]

    T., Norman, K

    Franklin, N. T., Norman, K. A., Ranganath, C., Zacks, J. M., and Gershman, S. J. Structured event memory: A neuro-symbolic model of event cognition. Psychological Review, 127 0 (3): 0 327--361, 2020. doi:10.1037/rev0000177

    2020 doi

  14. [22]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2024

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., Chen, P., Li, Y., Lin, S., Zhao, S., Li, K., Xu, T., Zheng, X., Chen, E., Ji, R., and Sun, X. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in vide...

    2024

  15. [23]

    O., and Nader, K

    Hardt, O., Einarsson, E. O., and Nader, K. A bridge over troubled water: reconsolidation as a link between cognitive and neuroscientific memory research traditions. Annual Review of Psychology, 61: 0 141--167, 2010. doi:10.1146/annurev.psych.093008.100455

    2010

  16. [24]

    Langchain, 2023

    hwchase17. Langchain, 2023. URL https://github.com/hwchase17/langchain. Accessed: 2023-12-20

    2023

  17. [25]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023

    2023

  18. [26]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Jin, P., Takanobu, R., Zhang, C., Cao, X., and Yuan, L. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08046, 2023

    2023 arXiv

  19. [27]

    BLIP -2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. BLIP -2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023 a

    2023

  20. [28]

    Videochat: Chat-centric video understanding

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., and Qiao, Y. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023 b

    2023 arXiv

  21. [29]

    Unmasked teacher: Towards training-efficient video foundation models

    Li, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L., and Qiao, Y. Unmasked teacher: Towards training-efficient video foundation models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 19891--19903, 2023 c

    2023

  22. [30]

    Mvbench: A comprehensive multi-modal video understanding benchmark, 2024

    Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., Wang, L., and Qiao, Y. Mvbench: A comprehensive multi-modal video understanding benchmark, 2024. URL https://arxiv.org/abs/2311.17005

    2024 arXiv

  23. [31]

    Llama-vid: An image is worth 2 tokens in large language models, 2023 d

    Li, Y., Wang, C., and Jia, J. Llama-vid: An image is worth 2 tokens in large language models, 2023 d

    2023

  24. [32]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023

    2023

  25. [33]

    Kangaroo: A powerful video-language model supporting long-context video input, 2024 a

    Liu, J., Wang, Y., Ma, H., Wu, X., Ma, X., Wei, X., Jiao, J., Wu, E., and Hu, J. Kangaroo: A powerful video-language model supporting long-context video input, 2024 a

    2024

  26. [34]

    St-llm: Large language models are effective temporal learners, 2024 b

    Liu, R., Li, C., Tang, H., Ge, Y., Shan, Y., and Li, G. St-llm: Large language models are effective temporal learners, 2024 b

    2024

  27. [35]

    Nvila: Efficient frontier visual language models, 2024 c

    Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S., Xi, H., Cao, S., Gu, Y., Li, D., Li, X., Fang, Y., Chen, Y., Hsieh, C.-Y., Huang, D.-A., Cheng, A.-C., Nath, V., Hu, J., Liu, S., Krishna, R., Xu, D., Wang, X., Molchanov, P., Kautz, J., Yin, H., Han, S., and Lu, Y. Nvil...

    2024

  28. [36]

    Valley: Video assistant with large language model enhanced ability, 2023

    Luo, R., Zhao, Z., Yang, M., Dong, J., Qiu, M., Lu, P., Wang, T., and Wei, Z. Valley: Video assistant with large language model enhanced ability, 2023

    2023

  29. [37]

    Changing concepts of working memory

    Ma, W., Husain, M., and Bays, P. Changing concepts of working memory. Nature Neuroscience, 17: 0 347--356, 2014. doi:10.1038/nn.3655. URL https://doi.org/10.1038/nn.3655

    2014 doi

  30. [38]

    Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024

    2024

  31. [39]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Mangalam, K., Akshulakov, R., and Malik, J. Egoschema: A diagnostic benchmark for very long-form video language understanding. arXiv preprint arXiv:2308.09126, 2023

    2023 arXiv

  32. [40]

    Sparse and continuous attention mechanisms

    Martins, A., Farinhas, A., Treviso, M., Niculae, V., Aguiar, P., and Figueiredo, M. Sparse and continuous attention mechanisms. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 20989...

    2020

  33. [41]

    Martins, A. F. T., Treviso, M., Farinhas, A., Aguiar, P. M. Q., Figueiredo, M. A. T., Blondel, M., and Niculae, V. Sparse continuous distributions and fenchel-young losses. Journal of Machine Learning Research, 23 0 (257): 0 1--74, 2022 a . URL http://jmlr.org/papers/v23/21-0879.html

    2022

  34. [42]

    H., Marinho, Z., and Martins, A

    Martins, P. H., Marinho, Z., and Martins, A. F. -former: Infinite memory transformer. In Proc. ACL, 2022 b

    2022

  35. [43]

    Making lasting memories: Remembering the significant

    McGaugh, J. Making lasting memories: Remembering the significant. Proceedings of the National Academy of Sciences of the United States of America, 110: 0 10402--10407, 2013. doi:10.1073/pnas.1301209110. URL https://doi.org/10.1073/pnas.1301209110

    2013 doi

  36. [44]

    McNamee, D. C. The generative neural microdynamics of cognitive processing. Current Opinion in Neurobiology, 85: 0 102855, 2024. doi:10.1016/j.conb.2024.102855. Epub 2024 Feb 29

    2024

  37. [45]

    OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapi...

    2024

  38. [46]

    Preston, A. R. and Eichenbaum, H. The interplay of hippocampus and prefrontal cortex in memory-based decision making. Current Biology, 23 0 (17): 0 R764--R773, 2013

    2013

  39. [47]

    Radvansky, G. A. and Zacks, J. M. Event Cognition. Oxford University Press, New York, NY, 2014

    2014

  40. [48]

    Video-xl: Extra-long vision language model for hour-scale video understanding, 2024

    Shu, Y., Liu, Z., Zhang, P., Qin, M., Zhou, J., Liang, Z., Huang, T., and Zhao, B. Video-xl: Extra-long vision language model for hour-scale video understanding, 2024

    2024

  41. [49]

    Moviechat: From dense token to sparse memory for long video understanding

    Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Guo, X., Ye, T., Lu, Y., Hwang, J.-N., et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449, 2023

    2023 arXiv

  42. [50]

    Moviechat+: Question-aware sparse memory for long video question answering

    Song, E., Chai, W., Ye, T., Hwang, J.-N., Li, X., and Wang, G. Moviechat+: Question-aware sparse memory for long video question answering. arXiv preprint arXiv:2404.17176, 2024

    2024 arXiv

  43. [51]

    Energy and policy considerations for deep learning in NLP

    Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in NLP . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 3645--3650, Florence, Italy, July 2019. Association for Computational Linguis...

    2019 doi

  44. [52]

    Episodic memory: From mind to brain

    Tulving, E. Episodic memory: From mind to brain. Annual Review of Psychology, 53: 0 1--25, 2002. doi:10.1146/annurev.psych.53.100901.135114. URL https://doi.org/10.1146/annurev.psych.53.100901.135114

    2002

  45. [53]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Informatio...

    2017

  46. [54]

    Videoagent: Long-form video understanding with large language model as agent

    Wang, X., Zhang, Y., Zohar, O., and Yeung-Levy, S. Videoagent: Long-form video understanding with large language model as agent. European Conference on Computer Vision (ECCV), 2024 a

    2024

  47. [55]

    Videollamb: Long video understanding with recurrent memory bridges

    Wang, Y., Xie, C., Liu, Y., and Zheng, Z. Videollamb: Long video understanding with recurrent memory bridges. arxiv, 2024 b

    2024

  48. [56]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos

    Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., and Bansal, M. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. arXiv preprint arXiv:2405.19209, 2024 c

    2024 arXiv

  49. [57]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Xiao, J., Shang, X., Yao, A., and Chua, T.-S. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 9777--9786, 2021

    2021

  50. [58]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024

    Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., and Zhou, J. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024

    2024

  51. [59]

    M., Wang, Z., Yu, S., Bansal, M., and Bertasius, G

    Zhang, C., Lu, T., Islam, M. M., Wang, Z., Yu, S., Bansal, M., and Bertasius, G. A simple llm framework for long-range video question-answering, 2023 a

    2023

  52. [60]

    Videollama: An instruction-tuned audio-visual language model for video understanding

    Zhang, H., Li, X., and Bing, L. Videollama: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), 2023 b

    2023

  53. [61]

    Long context transfer from language to vision

    Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., and Liu, Z. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. URL https://arxiv.org/abs/2406.16852

    2024 arXiv

  54. [62]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.