Pith. sign in

REVIEW 5 major objections 4 minor 7 cited by

Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding

T0 review · 5 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that adding a query-guided temporal working memory module, without retraining the backbone, improves nine multimodal models on audio-visual question answering, video captioning, and video-text retrieval.

desk verdict A useful engineering idea with broad evaluation, but the undefined scoring rule and missing random-selection baseline leave the central claim unproven. read the letter →

arxiv 2502.06020 v1 pith:K4WICMQW submitted 2025-02-09 cs.CV cs.MMcs.SDeess.AS

Xingjian Diao , Chunhui Zhang , Weiyi Wu , Zhongyu Ouyang , Peijun Qing , Ming Cheng , Soroush Vosoughi , Jiang Gui This is my paper · ORCID
classification cs.CVcs.MMcs.SDeess.AS
keywords temporalworkingmemoryquery-guidedattentionmultimodalfoundationmodelsaudio-visualquestionansweringvideocaptioningvideo-textretrievalsegmentselectionreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Temporal Working Memory (TWM), a plug-and-play module that selects the most query-relevant video frames and audio segments before they are fed into a multimodal foundation model (MFM). The authors claim that integrating TWM into nine state-of-the-art MFMs produces consistent gains on audio-visual question answering (MUSIC-AVQA v2.0), video captioning (MSR-VTT), and video-text retrieval (CMD). The motivation is that MFMs have finite internal capacity and currently process long temporal sequences indiscriminately, so selectively retaining relevant segments should improve temporal reasoning. If correct, TWM offers a model-agnostic way to extend MFMs to longer, time-sensitive video and audio inputs.

What carries the argument

The load-bearing object is the query-guided neural search engine with scoring rule $S(v_i) = \alpha_1 D(v_i) + \alpha_2 R(v_i, q)$. It is trained with InfoNCE loss to align visual and audio embeddings with text-query embeddings, then iteratively selects the most relevant frames within a search window; the same routine, using visual embeddings as queries, maintains an auditory buffer. The mechanism also uses inter-segment and intra-segment attention over audio segments to model both long-range dependencies and fine-grained local changes.

What would settle it

Run each baseline with the same number of selected frames chosen uniformly at random instead of by the $S(v_i)$ score, keeping everything else fixed; if random selection matches TWM's gains, the query-guided scoring is not the active ingredient. Additionally, replacing $R(v_i,q)$ with a constant or with random relevance scores should change performance if query guidance matters.

Watch

Extended reading notes

Core claim

TWM alternates between search and update operations: it scores each video frame with $S(v_i) = \alpha_1 D(v_i) + \alpha_2 R(v_i, q)$, where $D$ measures distinctiveness and $R$ measures query relevance, and iteratively selects frames in a window around the highest-scoring midpoint while excluding already-selected frames. Audio is divided into segments, aligned to visual embeddings through InfoNCE-trained cross-modal similarity, and refined by inter-segment and intra-segment attention to capture both global and local temporal structure. The selected segments form a temporal memory buffer that replaces the full sequence as the MFM input. The paper reports that this mechanism yields accuracy and retrieval improvements across all nine baselines, with the largest gains in comparative audio-visual question answering and captioning coherence.

Load-bearing premise

The load-bearing premise is that the weighted sum of a frame's distinctiveness and its cosine-similarity relevance to the question, $S(v_i)=\alpha_1 D(v_i)+\alpha_2 R(v_i,q)$, is a reliable proxy for how much that frame or audio segment helps the downstream model, so if the proxy is wrong the gains could come from subsampling rather than from query-guided memory.

Editorial extensions

If this is right

  • Adding TWM to an existing multimodal model improves audio-visual question answering accuracy, especially on comparative audio and audio-visual questions where several baselines gain 10 to 13 percentage points.
  • Video captioning becomes more temporally coherent, with Git+TWM gaining 6.82 CIDEr points on MSR-VTT while using far fewer frames.
  • Video-text retrieval improves across Recall@1, Recall@5, and Recall@10 for VINDLU, TESTA, and MovieSeq on the CMD benchmark.
  • TWM reduces the number of frames processed per video, so it uses the model's limited capacity more efficiently without changing the backbone architecture.
  • TWM can be applied to visual-only, audio-only, or audio-visual pipelines by omitting the auditory component when audio is not present.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains may partly come from aggressive frame subsampling itself rather than from query-guided selection; a random-selection control with the same number of frames would separate these effects, and the paper does not report one.
  • The scoring functions $D$ and $R$ are left unspecified, so a sharp test would be to define them precisely and ablate the query-relevance term; if performance is insensitive to $R(v_i,q)$, the query guidance is not the active ingredient.
  • TWM resembles token-pruning and frame-sampling methods in video-language models, so connecting it to those established lines could clarify whether the working-memory framing adds predictive power beyond existing sampling techniques.
  • If the mechanism works as claimed, a natural extension the authors leave implicit is online or streaming operation, where the memory buffer is updated as new frames and audio arrive rather than selected once from a fixed sequence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes Temporal Working Memory (TWM), a plug-and-play module for multimodal foundation models (MFMs) that selects a small number of query-relevant video frames and audio segments from long multimodal inputs. The module uses a query-guided scoring rule to iteratively fill visual and auditory buffers, which are then fed to the downstream MFM. The authors integrate TWM into nine existing MFMs and report accuracy/recall scores on MUSIC-AVQA v2.0, MSR-VTT, and CMD, claiming consistent performance improvements across all tasks and models. The paper also includes ablation studies, case-study visualizations, and a link to released code.

Significance. If the claimed gains are real, TWM would provide a lightweight, model-agnostic way to improve temporal reasoning in MFMs without changing the backbone or retraining it. The idea of using a query-guided memory buffer, analogous to human working memory, is plausible and timely, given the context-length limitations of current MFMs. The paper ships code and integrates with multiple open-source baselines, which is a strength. However, the current evidence does not conclusively isolate the effect of query-guided selection from the effect of subsampling itself, and the central scoring rule is not fully specified. These gaps are fixable with additional experiments and clarifications.

major comments (5)
  1. [Section 3.1.1, Eq. (1)] The functions D(vi) and R(vi, q) in Eq. (1) are never defined. The text calls them 'distinctiveness' and 'relevance to the query' but gives no formulas, no encoders, no input representations, and no computational details. Since Eq. (1) is the core scoring rule that drives the entire selection process, this omission prevents reproduction and makes it impossible to assess what the mechanism actually computes. Please specify D and R explicitly (e.g., as functions of the InfoNCE-aligned embeddings, maybe with normalization or margins), or at minimum provide a complete pseudocode for sim and select in Algorithm 1.
  2. [Section 4.1 and 4.2] No random or uniform frame/audio selection baseline is reported. The reported gains could arise from subsampling benefits (e.g., fewer tokens reducing noise or computational load) rather than from query-guided selection. Given that k, the number of iterations, and alpha1/alpha2 are tuned per dataset (Section 4.1), a control that keeps all hyperparameters identical but selects frames uniformly at random (or with D=0, R=0) is necessary to attribute the improvements to query-guided temporal memory. Please add this control for at least one or two representative baselines per task.
  3. [Table 1] Several per-category accuracies decrease even where the overall average improves. For example, LAVisH+TWM drops 4.60 on Audio Count and 8.24 on Audio-Visual Exist; DG-SCT+TWM drops 3.08 on Audio Count and 7.52 on Audio-Visual Temporal; LAST-Att+TWM drops 6.51 on Audio Count and 3.01 on Audio Average. The abstract and Section 4.2 claim 'significant performance improvements across all tasks', which is stronger than what the data show. Also, all tables report point estimates without error bars or significance tests, so it is unclear which of the positive gains are statistically reliable. Please temper the claim and add variance estimates or paired significance tests.
  4. [Appendix B] The ablations remove entire visual or auditory components (TWM w/o VWM, TWM w/o AWM), but they do not test whether the query-guided selection rule itself matters. A more informative ablation would keep the same memory size and iterative procedure but replace the relevance-based scoring in Eq. (1) with random selection, to isolate the contribution of query-guided scoring. As it stands, the ablations show that both visual and auditory components are needed, but not that the query-guided search is the reason for the gains.
  5. [Section 3.1.2] The training of the neural search engine is underspecified. It says a linear projection layer is trained with InfoNCE to align visual embeddings with text, but it does not state which dataset is used for this training, whether the projection is trained jointly with the downstream MFM or separately, or which text descriptions serve as positives/negatives for a given video. This affects reproducibility. Please clarify the training protocol, including the exact data source and the negative sampling scheme.
minor comments (4)
  1. [Section 3.2.1] Audio segmentation is described as 'typically 5-6 segments depending on video length', but the implementations section (Section 4.1) says 'Audio segments are extracted every 5 seconds, selecting the highest-scoring segment from a total of 12 segments'. Please reconcile these numbers and specify the segmentation strategy consistently.
  2. [Figure 6] The figure captions refer to Frame 0, Frame 72, etc., but the frames are not shown in chronological order in the first row (Frame 0, 72, 108, 116, 125, 148, 180, 249); please make the ordering explicit so readers can follow the temporal progression.
  3. [Section 4.2.1] The text says 'LAVisH+TWM improves by 12.40%' for audio-related comparative QA, but Table 1 shows the gain is +12.40 for Audio Compare; the percentage sign is redundant but not wrong. More importantly, the same paragraph says 'significant increases' without any significance testing; please replace 'significant' with 'numerical' unless a statistical test is provided.
  4. [References] There is an inconsistency in the references for LLaMA: both Touvron et al. 2023a and 2023b are listed with the same arXiv number (2302.13971); the LLaMA reference should be corrected to the actual model paper. Also, the 'Git' method is referred to as 'Git' in Table 2 but as 'Git (Wang et al., 2022)' in the text; please be consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TWM's reported gains are empirical benchmark comparisons, not derived from fitted equations or from load-bearing self-citations.

full rationale

The paper's derivation chain is empirical rather than formal: TWM is defined as a query-guided segment selector (Eq. (1), Algorithm 1), and its contribution is measured by comparing nine MFMs with and without the module on MUSIC-AVQA v2.0, MSR-VTT, and CMD (Tables 1-3). Eq. (1) is an implementation rule for ranking segments, not a relation fitted to the reported outcome; D and R are underspecified (never given explicit formulas), and Algorithm 1's sim/select are placeholders, but the paper does not derive the benchmark gains from Eq. (1), so there is no equation-level identity between input and output. The InfoNCE losses in Eqs. (2)-(3) train the alignment used by the selector; because evaluation uses external benchmark test sets, the reported gains are not forced by the training objective. Self-citations (Zhang et al. 2024; Diao et al. 2024; Zhang et al. 2025) appear only in background and related-work context and are not load-bearing; no uniqueness claim or ansatz is imported from the authors' prior work. The undefined scoring functions, the absence of a random-selection control, and the possibility of per-dataset hyperparameter tuning are reproducibility and validity concerns, not circularity; the Limitations section itself flags only unexplored generalizability.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The central claim depends on the assumption that pretrained encoder similarities are reliable proxies for task-relevant content, on the undefined D and R functions in Eq. (1), and on per-dataset hyperparameters chosen manually. The module itself is validated only on the paper's benchmarks.

free parameters (6)
  • alpha1 (frame distinctiveness weight) = 0.2 (MUSIC-AVQA), 0.5 (MSR-VTT), 0.6 (CMD)
    Used in Eq. (1) to weight distinctiveness D(vi); set per dataset in Section 4.1, with no sensitivity analysis.
  • alpha2 (query relevance weight) = 0.8 (MUSIC-AVQA), 0.5 (MSR-VTT), 0.4 (CMD)
    Used in Eq. (1) to weight query relevance R(vi,q); set per dataset in Section 4.1.
  • k (frames selected per search step) = 11 (MUSIC-AVQA), 3 (MSR-VTT), 5 (CMD)
    Controls the number of frames considered per iteration; manually chosen per dataset in Section 4.1.
  • number of search iterations = 6 (MUSIC-AVQA), 3 (MSR-VTT), 7 (CMD)
    Controls the total selected frame count; manually chosen per dataset in Section 4.1.
  • InfoNCE temperature tau = not specified
    Temperature in Eqs. (2) and (3) is not reported.
  • audio segment length and count = 5-second segments; best of 12 for MUSIC-AVQA; 5-6 segments typical otherwise
    Audio segmentation granularity is chosen per dataset and directly affects the auditory buffer.
assumptions (4)
  • domain assumption Cosine similarity between pretrained visual, language, and audio embeddings, after InfoNCE alignment, reflects how relevant a frame or audio segment is to the query.
    Sections 3.1.2 and 3.2.2 train linear projections with InfoNCE and use the resulting similarities to select segments; no external validation of this proxy is provided.
  • ad hoc to paper The functions D(vi) and R(vi,q) in Eq. (1) are well-defined and computable.
    Section 3.1.1 defines S(vi) with these functions but never gives their formulas.
  • domain assumption Aggressive subsampling, for example 8-9 frames from 210-630, preserves the information needed by the downstream MFM.
    Section 4.3 argues selected frames capture key stages, but no systematic comparison with all frames or random frames is provided.
  • domain assumption Baseline models are reproduced faithfully with open-source code and pretrained weights.
    Appendix A lists baselines; no code or hash is supplied to verify the reproductions.
invented entities (1)
  • Temporal Working Memory (TWM) module with visual and auditory buffers
    purpose: Retains query-relevant video frames and audio segments before MFM processing
    The module is evaluated only on the paper's own benchmark setups; there is no external falsifiable handle such as a formal proof, an independent dataset prediction, or released code with a commit hash.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding." pith.science (2026). https://pith.science/paper/K4WICMQW

@misc{pith2026250206020,
  author       = {Pith},
  title        = {Pith review of: Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K4WICMQW}},
  note         = {Machine review of arXiv:2502.06020}
}
read the original abstract

Multimodal foundation models (MFMs) have demonstrated significant success in tasks such as visual captioning, question answering, and image-text retrieval. However, these models face inherent limitations due to their finite internal capacity, which restricts their ability to process extended temporal sequences, a crucial requirement for comprehensive video and audio analysis. To overcome these challenges, we introduce a specialized cognitive module, temporal working memory (TWM), which aims to enhance the temporal modeling capabilities of MFMs. It selectively retains task-relevant information across temporal dimensions, ensuring that critical details are preserved throughout the processing of video and audio content. The TWM uses a query-guided attention approach to focus on the most informative multimodal segments within temporal sequences. By retaining only the most relevant content, TWM optimizes the use of the model's limited capacity, enhancing its temporal modeling ability. This plug-and-play module can be easily integrated into existing MFMs. With our TWM, nine state-of-the-art models exhibit significant performance improvements across tasks such as video captioning, question answering, and video-text retrieval. By enhancing temporal modeling, TWM extends the capability of MFMs to handle complex, time-sensitive data effectively. Our code is available at https://github.com/xid32/NAACL_2025_TWM.

Figures

Figures reproduced from arXiv: 2502.06020 by the authors.

Figure 1
Figure 1. Temporal Working Memory (TWM): TWM employs search engine and memory refresh mechanisms to retain key segments in long multimodal inputs. et al., 2024a; Xie et al., 2024; Yao et al., 2024b; Xie et al., 2025; Han et al., 2024; Liu et al., 2024d; Lin et al., 2024). While MFMs excel at processing multimodal inputs, MFMs are often not equipped to explicitly reduce the input context burden, particu￾larly in extracting que… view at source ↗
Figure 2
Figure 2. The temporal working memory (TWM) pipeline retains the most relevant segments from video and audio [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Aligning frames with language query. A linear [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Audio segments aligned with query-relevant [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: TWM-searched frames for the video captioning task, integrated with AKGNN ( [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Ablation studies on different components of TWM using MUSIC-AVQA v2.0 dataset ( [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: TWM on video-text retrieval models using [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: TWM on video captioning models using MSR [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: TWM-searched frames with corresponding questions for audio-visual question answering, integrated with [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: TWM-searched frames with corresponding questions for audio-visual question answering, integrated [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: TWM-searched frames with corresponding text description for video-text retrieval, integrated with [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: TWM-searched frames with corresponding text description for video-text retrieval, integrated with [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Language-Guided Long Horizon Manipulation with LLM-based Planning and Visual Perception

    cs.RO 2025-09 conditional novelty 5.0 of 10

    A robot folds cloth from spoken language by decomposing instructions with GPT-4o and grounding each step with a SigLIP2-based pick-and-place perception module.

  2. GDLLM: A Global Distance-aware Modeling Approach Based on Large Language Models for Event Temporal Relation Extraction

    cs.CL 2025-08 conditional novelty 5.0 of 10

    GDLLM improves event temporal relation extraction by feeding LLM-generated probability distributions into a graph attention network, achieving state-of-the-art micro-F1 scores on TB-Dense and MATRES.

  3. FakeSV-VLM: Taming VLM for Detecting Fake Short-Video News via Progressive Mixture-Of-Experts Adapter

    cs.MM 2025-08 reject novelty 5.0 of 10

    FakeSV-VLM reaches 90.22% and 89.30% accuracy on FakeSV and FakeTT by adding a two-stage MoE adapter and contrastive alignment to InternVL2.5-8B.

  4. OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning

    cs.CV 2025-09 conditional novelty 4.0 of 10

    OpenVision 2 shows that a caption-only generative objective can match contrastive learning for multimodal vision encoders at lower training cost, scaling to 1B parameters.

  5. Learning Sparsity for Effective and Efficient Music Performance Question Answering

    cs.SD 2025-06 conditional novelty 4.0 of 10

    Sparsify reports state-of-the-art accuracy on Music AVQA benchmarks by borrowing three existing sparsification techniques, cutting training time by 28% and retaining 70-80% of accuracy on a 25% data subset.

  6. A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecture

    cs.LG 2025-09 reject novelty 3.0 of 10

    The paper claims a BiLSTM-AM-VMD model achieves AUC 0.963 for early HCC diagnosis, but the evidence is undermined by contradictory dataset descriptions and missing artifacts.

  7. Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkers

    cs.LG 2025-09 reject novelty 3.0 of 10

    XGBoost combining MRI radiomics and clinical biomarkers reportedly reaches C-index 0.782 for early brain tumor recurrence, but the paper's methods describe a liver-cancer cohort and no evaluation of its claimed tempor...

Reference graph

Works this paper leans on

47 extracted references · 35 canonical work pages · cited by 7 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alan Baddeley. 2000. The episodic buffer: a new component of working memory? Trends in cognitive sciences

    work page 2000

  4. [4]

    Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. 2020. Condensed movies: Story based retrieval with contextual embeddings. In Asian Conference on Computer Vision

    work page 2020

  5. [5]

    Peng Chen, Yingying Zhang, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, and Chenjuan Guo. 2024. Pathformer: Multi-scale transformers with adaptive pathways for time series forecasting. In International Conference on Learning Representations

    work page 2024

  6. [6]

    Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius. 2023. Vindlu: A recipe for effective video-and-language pretraining. In Conference on Computer Vision and Pattern Recognition

    work page 2023

  7. [7]

    Xingjian Diao, Chunhui Zhang, Tingxuan Wu, Ming Cheng, Zhongyu Ouyang, Weiyi Wu, and Jiang Gui. 2024. Learning musical representations for music performance question answering. In Findings of the Association for Computational Linguistics: EMNLP 2024

    work page 2024

  8. [8]

    Haoyi Duan, Yan Xia, Zhou Mingze, Li Tang, Jieming Zhu, and Zhou Zhao. 2023. Cross-modal prompts: Adapting large pre-trained models for audio-visual downstream tasks. In Advances in Neural Information Processing Systems

    2023

Show all 47 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Dongyu Gong, Xingchen Wan, and Dingmin Wang. 2024. Working memory capacity of chatgpt: An empirical study. In AAAI Conference on Artificial Intelligence

    2024

  3. [11]

    Albert Gu, Karan Goel, and Christopher R \'e . 2022. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations

    2022

  4. [12]

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. 2024. Onellm: One framework to align all modalities with language. In Conference on Computer Vision and Pattern Recognition

    2024

  5. [13]

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. 2024. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Conference on Computer Vision and Pattern Recognition

    2024

  6. [14]

    Willy Fitra Hendria, Vania Velda, Bahy Helmi Hartoyo Putra, Fikriansyah Adzaka, and Cheol Jeong. 2023. Action knowledge for video captioning with graph neural networks. Journal of King Saud University-Computer and Information Sciences

    2023

  7. [15]

    Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. 2019. Timeception for complex action recognition. In Conference on Computer Vision and Pattern Recognition

    2019

  8. [16]

    Md Mohaiminul Islam and Gedas Bertasius. 2022. Long movie clip classification with state-space video models. In European Conference on Computer Vision

    2022

  9. [17]

    Yiren Jian, Tingkai Liu, Yunzhe Tao, Chunhui Zhang, Soroush Vosoughi, and Hongxia Yang. 2024. Expedited training of visual conditioned language generation via redundancy reduction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics

    2024

  10. [18]

    Byoungjip Kim, Dasol Hwang, Sungjun Cho, Youngsoo Jang, Honglak Lee, and Moontae Lee. 2024. Show think and tell: Thought-augmented fine-tuning of large language models for video captioning. In Conference on Computer Vision and Pattern Recognition

    2024

  11. [19]

    Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2023 a . Large language models with controllable working memory. In Findings of the Association for Computational Linguistics: ACL

    2023

  12. [20]

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Conference on Computer Vision and Pattern Recognition

    2022

  13. [21]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 b . BLIP -2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning

    2023

  14. [22]

    Kevin Qinghong Lin, Pengchuan Zhang, Difei Gao, Xide Xia, Joya Chen, Ziteng Gao, Jinheng Xie, Xuhong Xiao, and Mike Zheng Shou. 2024. Learning video context as interleaved multimodal sequences. In European Conference on Computer Vision

    2024

  15. [23]

    Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. 2023. Vision transformers are parameter-efficient audio-visual learners. In Conference on Computer Vision and Pattern Recognition

    2023

  16. [24]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems

    2023

  17. [25]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 a . Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics

    2024

  18. [26]

    Xiulong Liu, Zhikang Dong, and Peng Zhang. 2024 b . Tackling data bias in music-avqa: Crafting a balanced dataset for unbiased question-answering. In Winter Conference on Applications of Computer Vision

    2024

  19. [27]

    Zheyuan Liu, Guangyao Dou, Eli Chien, Chunhui Zhang, Yijun Tian, and Ziwei Zhu. 2024 c . Breaking the trilemma of privacy, utility, and efficiency via controllable machine unlearning. In International World Wide Web Conference

    2024

  20. [28]

    Zheyuan Liu, Guangyao Dou, Mengzhao Jia, Zhaoxuan Tan, Qingkai Zeng, Yongle Yuan, and Meng Jiang. 2024 d . Protecting privacy in multimodal large language models with mllmu-bench. arXiv preprint arXiv:2410.22108

    2024 arXiv

  21. [29]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. 2024. Video- C hat GPT : Towards detailed video understanding via large vision and language models. In Annual Meeting of the Association for Computational Linguistics

    2024

  22. [30]

    Asmar Nadeem, Faegheh Sardari, Robert Dawes, Syed Sameed Husain, Adrian Hilton, and Armin Mustafa. 2024. Narrativebridge: Enhancing video captioning with causal-temporal narrative. arXiv preprint arXiv:2406.06499

    2024 arXiv

  23. [31]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

    2018 arXiv

  24. [32]

    Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, and Lu Hou. 2023. TESTA : Temporal-spatial token aggregation for long-form video-language understanding. In Findings of the Association for Computational Linguistics: EMNLP

    2023

  25. [33]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  26. [34]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  27. [35]

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. Git: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research

    2022

  28. [36]

    Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. 2023. Selective structured state-spaces for long-form video understanding. In Conference on Computer Vision and Pattern Recognition

    2023

  29. [37]

    Chao-Yuan Wu and Philipp Krahenbuhl. 2021. Towards long-form video understanding. In Conference on Computer Vision and Pattern Recognition

    2021

  30. [38]

    Penghao Wu and Saining Xie. 2024. V*: Guided visual search as a core mechanism in multimodal llms. In Conference on Computer Vision and Pattern Recognition

    2024

  31. [39]

    Wulin Xie, Xiaohuan Lu, Yadong Liu, Jiang Long, Bob Zhang, Shuping Zhao, and Jie Wen. 2024. Uncertainty-aware pseudo-labeling and dual graph driven network for incomplete multi-view multi-label classification. In International Conference on Multimedia

    2024

  32. [40]

    Wulin Xie, Lian Zhao, Jiang Long, Xiaohuan Lu, and Bingyan Nie. 2025. Multi-view factorizing and disentangling: A novel framework for incomplete multi-view multi-label classification. arXiv preprint arXiv:2501.06524

    2025 arXiv

  33. [41]

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Conference on Computer Vision and Pattern Recognition

    2016

  34. [42]

    Jiawei Yao, Qi Qian, and Juhua Hu. 2024 a . Customized multiple clustering via multi-modal subspace proxy learning. In Advances in Neural Information Processing Systems

    2024

  35. [43]

    Jiawei Yao, Qi Qian, and Juhua Hu. 2024 b . Multi-modal proxy learning towards personalized visual multiple clustering. In Conference on Computer Vision and Pattern Recognition

    2024

  36. [44]

    Chunhui Zhang, Chao Huang, Youhuan Li, Xiangliang Zhang, Yanfang Ye, and Chuxu Zhang. 2022. Look twice as much as you say: Scene graph contrastive learning for self-supervised image caption generation. In International Conference on Information & Knowledge Management

    2022

  37. [45]

    Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi. 2024. Scaling cognitive limits: Identifying working memory limits in llms. In Conference on Empirical Methods in Natural Language Processing

    2024

  38. [46]

    Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi. 2025. Pretrained image-text models are secretly video captioners. In Annual Conference of the North American Chapter of the Association for Computational Linguistics

    2025

  39. [47]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Conference on Empirical Methods in Natural Language Processing

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.