Pith. sign in

REVIEW 5 major objections 4 minor 49 references

The paper claims that long-form video understanding is best served by a self-correcting, entity-centric memory that anchors identities to tracks, retroactively rewrites history when names arrive late, and abstains when evidence is missing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:41 UTC pith:A6ABCY76

load-bearing objection Plausible memory architecture with a genuinely interesting retroactive-rewrite idea, but the evidence quality is too thin to back the SOTA claims as written. the 5 major comments →

arxiv 2607.28678 v1 pith:A6ABCY76 submitted 2026-07-29 cs.AI cs.CV

ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understanding

Xinkui Zhao , Enbo Chen , Yifan Zhang , Chang Liu , Guanjie Cheng , Naibo Wang , Yueshen Xu This is my paper
classification cs.AI cs.CV
keywords multimodal memorylong-form video understandingentity-centric memoryself-correctionvision-language modelsretroactive refinementepistemic refusalidentity binding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that current memory-augmented video agents fail on hour-long videos because they lose identity cues under compression, process video in isolated segments so late identity evidence never reaches earlier events, and retrieve by semantic similarity, which mixes related-but-wrong entities together. To fix this, the authors propose ViSAGE, a memory framework with three parts: track-level entity anchoring (binding faces and voices to stable identities across the whole video), bidirectional refinement (retroactively unifying old incident logs with new name evidence), and multi-agent cross-verification (a judge that abstains when evidence doesn't align with identity). On three long-video benchmarks the framework reports state-of-the-art accuracy — 45.5% on M3-Bench-robot, 58.4% on M3-Bench-web, and 79.1% on Video-MME-long — about 5.9% higher than the strongest baseline overall. If right, this means retroactive, identity-first memory repair is a structural requirement, not a luxury, for long-horizon multimodal agents.

Core claim

The central claim is that identity must be treated as a first-class object in long-term multimodal memory. ViSAGE anchors each person to a track-level prototype (the best face instance on a long-range trajectory), binds speaker IDs to face IDs using both voice similarity and active-speaker detection, and stores events in two structures: append-only Incident Logs whose 'who' fields remain editable, and evolving Object Cards. When the language model later infers that a spoken name refers to a previously anonymous visual subject, it emits an equivalence signal and ViSAGE rewrites every historical log entry involving that subject, propagating the name backward in time. At query time, a dispatche

What carries the argument

The load-bearing object is the Dual-Track Memory Bank — Incident Logs (a chronological, append-only narrative where the who-field is mutable) paired with Object Cards (accumulating profiles of named entities). The two operations that make it self-correcting are (1) Sequence-Level Entity Anchoring, which converts per-frame identity guesses into stable track-to-prototype clusters and uses dual-criteria voice–face binding (acoustic similarity plus audio–visual synchrony), and (2) the Global Backward Update, which fires when the LLM concludes that a semantic identity (a name) is equivalent to a visual identity (a track), rewriting all prior Incident Log entries so late evidence repairs early mem

Load-bearing premise

The whole benefit rests on the LLM being right when it infers that a name in dialogue refers to a specific visible person; if that equivalence signal is wrong even occasionally, the global backward update rewrites the entire history around a false identity.

What would settle it

Construct a controlled benchmark where dialogue contains deliberately misleading cues — hearsay, sarcasm, or mispronounced names — and compare ViSAGE's final answers against a version of the same system with the backward update disabled. If the backward-update system turns a misattributed name into confident, historically consistent but wrong answers more often than the forward-only version, the paper's central claim fails. Additionally, measuring the precision and recall of the equivalence signals themselves (does a refinement trigger ever fire on an incorrect name-to-face mapping?) would set

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the reported numbers hold, long-horizon embodied agents can use aggressive video sampling (0.5 fps) and still recover identity-critical facts because late name evidence repairs early gaps, changing the cost-accuracy tradeoff in favor of cheaper preprocessing.
  • Verified refusals become a usable safety behavior: the judge abstains on unanswerable queries, which in embodied settings is safer than hallucinating an action-relevant answer.
  • The framework's model-agnostic dual-track design means the memory structure can be bolted onto different backbones; the paper reports consistent gains across weaker and stronger LLMs.
  • Retroactive rewriting makes the 'who' field of past events always consistent with the present, so downstream multi-hop reasoning about character attributes can be answered in one retrieval step instead of many.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves open is measuring the precision and recall of the refinement trigger itself — how often an inferred 'name → visual subject' equivalence is correct. Because the global update rewrites the whole history, a single systematic misattribution (e.g., from sarcasm or misheard names) could propagate into widespread corruption that a forward-only memory would not suffer; no su
  • The approach is human-centric by construction (face and voice identity); extending the same retroactive-repair logic to objects, animals, or places via open-vocabulary tracking is a natural generalization that the paper itself names as future work.
  • The paper's own limitation statement concedes that non-human entities are treated as background. If the central claim is about entity-centric memory in general, the current evidence only supports character-centric memory; a test on object-identity-heavy benchmarks would be needed.
  • One could synthesize a counterfactual stress test: take a video with two people who look alike or have similar voices, make the LLM misbind a name, and check whether the system confidently reports the wrong identity after refinement — that would directly probe the weakest assumption.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes ViSAGE, a multimodal agentic memory framework for long-form video understanding. The system maintains two memory structures, Incident Logs and Object Cards, and processes videos in three stages: entity-anchored preprocessing (face-track anchoring with voice-face binding), bidirectional memory refinement (forward identity-keyed retrieval plus a global backward update that rewrites prior Incident Logs when a name-to-face equivalence is inferred), and multi-agent cross-verification (dispatcher, scene/character agents, and a judge that can issue epistemic refusals). Experiments on M3-Bench-robot, M3-Bench-web, and Video-MME-long report state-of-the-art accuracy, with the largest gains attributed to the bidirectional refinement module. The authors argue that retroactive, entity-centric memory repair is a structural requirement for long-horizon video understanding.

Significance. If the reported results are reproducible, the paper makes a useful contribution: it moves beyond vector-similarity memory and chunk-wise processing by proposing a concrete, entity-centric memory architecture with retroactive correction. The framework is clearly described, the three components are intuitive, and the evaluation covers three benchmarks plus a cost analysis. The backbone-scaling experiment in Table 3 is a nice demonstration that the gains are not tied to a single MLLM. However, the empirical evidence as presented is not fully auditable: there are no error bars or significance tests, implementation thresholds are omitted, and, most importantly, the central backward-refinement mechanism is left without a false-trigger analysis even though it is an unconditional global rewrite. The safety and response-quality claims are also reported without methodological detail. The core idea is credible, but the current manuscript does not yet provide the evidence needed to support the strong 'structural necessity' and safety claims.

major comments (5)
  1. [§3.3, Fig. 4] The Global Backward Update is an unconditional rewrite: once an LLM produces a Refinement Signal (ID_sem ≡ ID_vis), the system 'traverses the historical Incident Logs L, retroactively replacing all prior instances' of the anonymous ID_vis. There is no confidence threshold, no verification step, and no rollback. If the name/addressee inference is wrong—due to pronoun ambiguity, sarcasm, overlapping speech, or ASR error—a single local error propagates to every historical record and to future retrieval. The paper reports no precision/recall for Refinement Signal triggers, no error taxonomy, and no analysis of the magnitude of corruption after a false trigger. This is load-bearing because the ablation in Fig. 4 shows that removing Bidirectional Memory Refinement causes the largest drop (−4.7% on robot), and Table 4 is only one hand-picked success. Without a false-positive analysis, the mecha
  2. [Table 1, §4.2] The headline '5.9% higher accuracy' is not representative: it matches only the robot split when ViSAGE (45.5) is compared with the strongest baseline Gemini-3-Pro (39.6). On M3-Bench-web the margin is 58.4 vs 53.8 = 4.6 points, and on Video-MME-long it is 79.1 vs 74.2 = 4.9 points. More importantly, no error bars, confidence intervals, or significance tests are reported. With only 100 videos in the robot split and 300 in Video-MME-long, a 4.6–5.9 point absolute difference may be within sampling noise. The authors should report multiple runs or bootstrap intervals, and where possible pairwise significance tests against the strongest baseline.
  3. [§4.4, Figs. 5–6] The safety and response-quality claims are not auditable as presented. The manuscript reports a 54.3% reduction in identity misalignment, a 31.0% reduction in hallucinations, a 264.3% increase in epistemic refusals, and win rates of 87.3%/91.1%/85.1% for visual richness, identity specification, and logical completeness. However, there are no definitions of the error categories, no sample sizes, no inter-annotator agreement, and no description of how the GPT-5 judge was prompted. If GPT-5 is used to evaluate a system whose own outputs are generated by LLM-based agents, the risk of judge bias needs to be addressed. The authors should provide the evaluation protocol, at least a sample of judged outputs, and ideally human validation on a subset.
  4. [§4.1, §3.2] Implementation details essential for reproducibility are missing. The paper does not specify the face-prototype match threshold, the voice-face binding thresholds used in Dual-Criteria Validation, the criterion for activating a Refinement Signal, the exact video sampling rate, or the prompt templates for the MLLM and the judge. These are free parameters that could materially affect the reported numbers. The authors should release code and configuration, or at minimum state the exact thresholds and prompts used, so that the results can be reproduced and the sensitivity of the main claim to these choices can be assessed.
  5. [§3.4, §4.4] The treatment of Verified Epistemic Refusal in the accuracy metric is unclear. The Judge Agent may abstain when evidence is insufficient, and Fig. 6 reports a large increase in refusals. If refusals are counted as incorrect answers, then accuracy should be penalized; if they are excluded or counted as correct, the comparison with baselines that always answer is not apples-to-apples. The paper must state how refusals were scored in the accuracy results, otherwise the numerical comparisons in Table 1 are ambiguous.
minor comments (4)
  1. [Abstract, §1] The abstract and introduction say '5.9% higher accuracy' without specifying the benchmark or comparison baseline; as noted, this margin applies only to M3-Bench-robot vs Gemini-3-Pro. Please state the comparison explicitly.
  2. [§4.1] Implementation details mention 'Qwen3-Omni-Flash' but the reference list contains 'Qwen3-Omni Technical Report'; align the naming and citation. Also, Table 1 formatting makes it hard to see which columns correspond to M3-Bench-robot vs M3-Bench-web; consider separating the sub-tables.
  3. [§3.3] The notation ID_sem ≡ ID_vis and the term 'Refinement Signal' are introduced informally. They should be defined more formally, including what input the LLM receives and what output format is expected, so that the update rule is unambiguous.
  4. [Figures 4–6] In the provided version, the figures appear to contain garbled path-like text; if this is a rendering artifact, ensure the final PDF displays the intended charts, since the ablation and safety numbers cannot be checked otherwise.

Circularity Check

0 steps flagged

No derivation-circularity: benchmark numbers come from external evaluations, and the framework modules are empirical designs rather than results derived from fitted inputs.

full rationale

I walked the paper's claimed derivation chain. There is no mathematical derivation or first-principles result: the paper reports an empirical system (ViSAGE) and compares its accuracy on M3-Bench-robot, M3-Bench-web, and Video-MME-long against baselines. The reported numbers (45.5%, 58.4%, 79.1%) are benchmark accuracies, not quantities computed from fitted parameters or defined in terms of the methods' outputs. The three mechanisms—entity-anchored preprocessing, bidirectional memory refinement, and multi-agent cross-verification—are algorithmic choices; none is defined in terms of the final benchmark scores. The ablation in Fig. 4 showing the largest drop when Bidirectional Memory Refinement is removed is an empirical observation, not a circular prediction. No self-citation is load-bearing: the paper does not cite its own prior work, and no uniqueness theorem or prior result by the same authors is invoked to force a conclusion. The closest self-referential element is the use of GPT-5 as a judge in the response-quality evaluation (Fig. 5), but that concerns subjective quality/safety ratings, not the headline accuracy numbers, and it is an evaluation-design choice rather than a derivation that reduces to its inputs. The skeptic's concern about the unconditional Global Backward Update propagating a single LLM misattribution is a real correctness/safety risk, and the absence of precision/recall analysis for Refinement Signals is a limitation, but it is not circularity: the mechanism's output is not equivalent to its input by construction. Under the stated criteria, no circular step can be exhibited with a specific equation or fitted-parameter reduction, so the score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim depends on the reliability of off-the-shelf perception tools, the reasoning of a proprietary MLLM, and the correctness of benchmark labels. These are domain assumptions rather than free parameters, but they are load-bearing. The explicit numerical choices (sampling rate, match thresholds) are not reported with sensitivity analysis. No new physical entities are introduced.

free parameters (4)
  • Face-prototype match threshold = not reported
    Controls whether degraded face observations are assigned to a trajectory's canonical identity in Section 3.2; mis-setting splits or merges identities, directly affecting Object Cards.
  • Voice-face binding thresholds = not reported
    Dual-Criteria Validation in Section 3.2 requires 'confidently identifiable with high voice-similarity' or 'ASD-confirmed synchrony'; no numerical thresholds are given.
  • Refinement signal activation = LLM-judged
    The global backward update is triggered by an LLM-generated 'Equivalence Signal' with no calibrated confidence threshold and no precision/recall analysis.
  • Video sampling rate = 0.5 fps
    Chosen by the authors to balance cost and fidelity; the paper reports no sensitivity analysis, though sampling rate directly affects identity cue availability.
axioms (4)
  • domain assumption Gemini-3-Pro and GPT-5 provide sufficiently reliable visual-linguistic inference for memory writing, retrieval, and judging.
    All modules rely on proprietary MLLM behavior; no public checkpoints or deterministic components are provided.
  • domain assumption Off-the-shelf multi-object tracking, face detection, voice diarization, and TalkNet active-speaker detection are accurate enough on the benchmarks.
    The identity anchoring of Section 3.2 depends on these tools; no sensitivity analysis or error analysis on downstream identity clusters is reported.
  • domain assumption Benchmark annotations and metrics correctly reward identity consistency and penalize hallucination.
    The accuracy claims treat benchmark labels as ground truth and do not independently measure entity consistency or refusal correctness.
  • domain assumption The LLM can reliably detect addressee and naming in dialogue without systematic bias.
    The refinement signal in Section 3.3 is generated by LLM inference with no false-positive control, which is load-bearing for the retroactive rewriting mechanism.

pith-pipeline@v1.3.0-alltime-deepseek · 17726 in / 12487 out tokens · 119612 ms · 2026-08-03T00:41:35.510695+00:00 · methodology

0 comments
read the original abstract

Multimodal agents operating in long-horizon environments must build and continually update multimedia memories to support entity-consistent, temporally grounded reasoning. However, existing agentic memory approaches often discard fine-grained dentity cues under aggressive compression and segment-wise processing. They also rely heavily on vector similarity retrieval, which can surface semantically related yet identity-mismatched evidence, leading to entity confusion, error propagation, and hallucinated answers. We propose ViSAGE, a multimodal agentic memory framework that constructs self-correcting, entity-centric memories. Specifically, ViSAGE anchors entity identity via cross-modal binding over long temporal ranges. It then applies bidirectional memory refinement to propagate delayed identity evidence, retroactively unifying historical records and improving future reasoning. We also introduce multi-agent cross-verification to assess retrieved evidence under an identity-evidence alignment onstraint, enabling abstention instead of unsupported answers when evidence is missing. Extensive results demonstrate that ViSAGE consistently outperforms the strongest baseline, achieving 5.9% higher accuracy.

Figures

Figures reproduced from arXiv: 2607.28678 by Chang Liu, Enbo Chen, Guanjie Cheng, Naibo Wang, Xinkui Zhao, Yifan Zhang, Yueshen Xu.

Figure 1
Figure 1. Figure 1: Superior long-horizon reasoning with ViSAGE. In complex long videos, standard memory struggles with de￾layed identities and audio-visual misalignment. ViSAGE ef￾fectively resolves these ambiguities. By establishing consis￾tent entity tracking across the timeline, it bypasses mislead￾ing noise and delivers precise, hallucination-free answers to complex temporal queries. streams, they often rely on aggressiv… view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of ViSAGE. The framework proceeds in three stages: (A) Entity-Anchoring Preprocessing binds visual and audio signals to annotated frames; (B) Bidirectional Context Refinement maintains self-correcting memory (Incident Logs and Object Cards) via retrieval and rectification; and (C) Epistemic Adjudication employs multi-agent cross-verification to ensure reliable responses. 2.2 Visual Context Exp… view at source ↗
Figure 3
Figure 3. Figure 3: Visualizing Bidirectional Memory Refinement. Late-appearing Identity Triggers activate a dual loop: Backward [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study on key components. The chart il [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Quality Assessment on Correct Samples. Even when [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Safety Analysis on M3-Bench-robot. ViSAGE signif￾icantly reduces hazardous hallucinations and increases safe epistemic refusals. this ambiguity. By retroactively grounding the first-person pro￾noun (“I”) to the target entity, our system enables the Judge Agent to prioritize direct testimony over third-person claims, effectively preventing hallucination. Beyond Accuracy: Response Granularity [PITH_FULL_IMA… view at source ↗
Figure 7
Figure 7. Figure 7: Cost and latency comparison among the Socratic [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 17 linked inside Pith

  1. [1]

    Shuai Bai et al. 2025. Qwen3-VL Technical Report. arXiv:2511.21631 [cs.CV] https://arxiv.org/abs/2511.21631 arXiv preprint arXiv:2511.21631

    Pith/arXiv arXiv 2025

  2. [2]

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, and Song Han. 2024. LongVILA: Scaling Long-Context Visual Language Models for Long Videos. https://arxiv.org/abs/2408.10188. arXiv preprint arXiv:2408.10188

    Pith/arXiv arXiv 2024

  3. [3]

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav

  4. [4]

    Yue Fan, Xiaojian Ma, Rongpeng Su, Jun Guo, Rujie Wu, Xi Chen, and Qing Li

  5. [5]

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. 2024. VideoAgent: A Memory-Augmented Multimodal Agent for Video Understanding. InProceedings of the European Conference on Computer Vision (ECCV). Springer, Cham, Switzerland, 75–92

    2024

  6. [6]

    InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Piscataway, NJ, USA, 6342–6352

  7. [7]

    Pascale Fung, Yoram Bachrach, Asli Celikyilmaz, Kamalika Chaudhuri, Delong Chen, Willy Chung, Emmanuel Dupoux, Hongyu Gong, Hervé Jégou, Alessandro Lazaric, et al. 2025. Embodied AI Agents: Modeling the World. https://arxiv.org/ abs/2506.22355. arXiv preprint arXiv:2506.22355

    Pith/arXiv arXiv 2025

  8. [8]

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2025. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA,...

    2025

  9. [9]

    Google DeepMind. 2025. Gemini 3: A New Era of Intelligence. https://deepmind. google/models/gemini/. Accessed: 2025-12-13

    2025

  10. [10]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context. https://arxiv.org/abs/2403.05530. arXiv preprint arXiv:2403.05530

    Pith/arXiv arXiv 2024

  11. [11]

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. 2024. MA-LMM: Memory-augmented Large Multimodal Model for Long-term Video Understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 13504–13514

    2024

  12. [12]

    Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, and Andrew Zisser- man. 2023. AutoAD: Movie Description in Context. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 18930–18940

    2023

  13. [13]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al

  14. [14]

    Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. 2025. HiAgent: Hierarchical Working Memory Management for Solving Long-horizon Agent Tasks with Large Language Models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL, Volume 1: Long Papers). Association for Computational Linguistics,...

    2025

  15. [15]

    Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. 2024. VidCompress: Memory- Enhanced Temporal Compression for Video Understanding in Large Language Models. https://arxiv.org/abs/2410.11417. arXiv preprint arXiv:2410.11417

    Pith/arXiv arXiv 2024

  16. [16]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al

  17. [17]

    Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Nagarajan, Lorenzo Torre- sani, and Gedas Bertasius. 2024. VideoRecap: Recursive Captioning of Hour-long Videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 18198–18208

    2024

  18. [18]

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. 2024. Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-horizon Tasks.Advances in Neural Information Processing Systems37 (2024), 49881–49913

    2024

  19. [19]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. Video-LLaVA: Learning United Visual Representation by Alignment Before Pro- jection. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Miami, Florida, USA, 5971–5984

    2024

  20. [20]

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. VILA: On Pre-training for Visual Language Models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 26689–26699

    2024

  21. [21]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2025. Videochat: Chat-centric video understanding. Science China Information Sciences68, 10 (2025), 200102

    2025

  22. [22]

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Liangwei Yang, Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, et al . 2024. AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent System. https://arxiv.org/abs/2402.15538. arXiv preprint arXiv:2402.15538

    Pith/arXiv arXiv 2024

  23. [23]

    Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan Lin, Hang Li, Junbo Zhao, and Wei Li. 2025. Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory. https://arxiv.org/abs/2508.09736. arXiv preprint arXiv:2508.09736

    arXiv 2025

  24. [24]

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Un- derstanding.Advances in Neural Information Processing Systems36 (2023), 46212– 46244

    2023

  25. [25]

    Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. 2025. Video- XL-Pro: Reconstructive Token Compression for Extremely Long Video Under- standing. https://arxiv.org/abs/2503.18478. arXiv preprint arXiv:2503.18478

    Pith/arXiv arXiv 2025

  26. [26]

    Kai Mei, Xi Zhu, Wujiang Xu, Wenyue Hua, Mingyu Jin, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. 2024. AIOS: LLM Agent Oper- ating System. https://arxiv.org/abs/2403.16971. arXiv preprint arXiv:2403.16971. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Zhao et al

    Pith/arXiv arXiv 2024

  27. [27]

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST). ACM, New York, NY, USA, 1–22

    2023

  28. [28]

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bor- des, et al. 2024. LongVU: Spatiotemporal Adaptive Compression for Long Video- Language Understanding. https://arxiv.org/abs/2410.17434. arXiv preprint arXiv:2410.17434

    Pith/arXiv arXiv 2024

  29. [29]

    Harry McGurk and John MacDonald. 1976. Hearing lips and seeing voices.Nature 264, 5588 (1976), 746–748

    1976

  30. [30]

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al . 2024. MovieChat: From Dense Token to Sparse Memory for Long Video Understand- ing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 18221–18232

    2024

  31. [31]

    Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. 2021. Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection. InProceedings of the 29th ACM International Conference on Multimedia. ACM, New York, NY, USA, 3927–3935

    2021

  32. [32]

    Endel Tulving. 2002. Episodic memory: From mind to brain.Annual review of psychology53, 1 (2002), 1–25

    2002

  33. [33]

    Singh, Adam Fry, Adam Perelman, Adam Tart, Adithya Ganesh, et al

    Aaditya K. Singh, Adam Fry, Adam Perelman, Adam Tart, Adithya Ganesh, et al

  34. [34]

    arXiv preprint arXiv:2601.03267

    OpenAI GPT-5 System Card. arXiv preprint arXiv:2601.03267

    Pith/arXiv arXiv

  35. [35]

    Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jiten- dra Malik, and Christoph Feichtenhofer. 2022. MemViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 13587–13597

    2022

  36. [36]

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. LongVideoBench: A Benchmark for Long-Context Interleaved Video-Language Understanding. Advances in Neural Information Processing Systems (NeurIPS)37 (2024), 28828– 28857

    2024

  37. [37]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. 2025. Qwen2. 5-omni technical report

    2025

  38. [38]

    Ying Wang, Yanlai Yang, and Mengye Ren. 2023. LifelongMemory: Leveraging LLMs for Answering Queries in Long-Form Egocentric Videos. https://arxiv.org/ abs/2312.05269. arXiv preprint arXiv:2312.05269

    Pith/arXiv arXiv 2023

  39. [39]

    Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. 2024. JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models.IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 46, 12 (2024), 8556–8571. doi:10.1109/TPAMI.2024.3391739

    arXiv 2024

  40. [40]

    Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. 2022. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language. https://arxiv.org/abs/2204.00598. arXiv preprint arXiv:2204.00598

    Pith/arXiv arXiv 2022

  41. [41]

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xi- aojie Jin. 2024. Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams. https://arxiv.org/abs/2406.08085. arXiv preprint arXiv:2406.08085

    Pith/arXiv arXiv 2024

  42. [42]

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Mem- oryBank: Enhancing Large Language Models with Long-Term Memory. InPro- ceedings of the AAAI Conference on Artificial Intelligence (AAAI), Vol. 38. AAAI Press, Washington, DC, USA, 19724–19731

    2024

  43. [43]

    Jin Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, Y. Wang, X. Shi, T. He, and X. Zhu

  44. [44]

    https://arxiv.org/abs/2509.17765

    Qwen3-Omni Technical Report. https://arxiv.org/abs/2509.17765. arXiv preprint arXiv:2509.17765

    Pith/arXiv arXiv

  45. [45]

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang

  46. [46]

    https://arxiv.org/abs/2502.12110

    A-Mem: Agentic Memory for LLM Agents. https://arxiv.org/abs/2502.12110. arXiv preprint arXiv:2502.12110

    Pith/arXiv arXiv

  47. [2020]

    9459– 9474 pages

    Retrieval-augmented generation for knowledge-intensive nlp tasks. 9459– 9474 pages

  48. [2024]

    https://arxiv.org/abs/2410.21276

    GPT-4o System Card. https://arxiv.org/abs/2410.21276. arXiv preprint arXiv:2410.21276

    Pith/arXiv arXiv

  49. [2025]

    https://arxiv.org/abs/2504.19413

    Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. https://arxiv.org/abs/2504.19413. arXiv preprint arXiv:2504.19413

    Pith/arXiv arXiv