Pith. sign in

REVIEW 4 major objections 5 minor 109 references

FOLIO: Focused Semantic Memory for Streaming Video Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Streaming video questions are best answered from a memory organized around persistent entities, written with focus-guided detail, rather than from compressed frames or event summaries alone, and FOLIO shows how to build and maintain that me

desk verdict A genuinely new entity-centered memory design for streaming video, but the headline Backward score is mostly a prompt artifact and needs to be re-reported before the SOTA claim is fair. read the letter →

arxiv 2607.13298 v1 pith:EHKUP24E submitted 2026-07-14 cs.CV

Haoyang Fan , Dhruv Parikh , Anvitha Ramachandran , Sameh Gobriel , Nilesh Jain , Rajgopal Kannan , Viktor Prasanna This is my paper · ORCID
classification cs.CV
keywords streamingvideounderstandingsemanticmemoryentitytrackingfocusstatecompressionretrieval-augmentedQAvision-languagemodelsmulti-turndialogue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FOLIO claims that the right unit of memory for streaming video understanding is the persistent entity, not the frame, token, scene, or event. As a video stream arrives, a dynamic focus state decides which entities and actions deserve detailed records and which can be kept compact, so the memory stays small while the information later queries need remains recoverable. The system is training-free: it uses an off-the-shelf vision-language model as a writer that produces structured entity records, merged into a long-term object ledger linked to cached keyframes. On OVO-Bench and StreamingBench, this focused entity memory improves accuracy over prior streaming-memory baselines while cutting writer input, output, and stored-memory size by roughly a third. A sympathetic reader would care because it suggests that memory cost and question-answering accuracy are not necessarily in conflict, if the memory is organized around the objects and actors that questions actually refer to.

What carries the argument

The central object is the entity-centered long-term semantic memory (the 'object ledger'), which stores each observed entity as a persistent slot with a canonical name, aliases, category, attributes, observation sequences, event sequences, and pointers into a visual-evidence cache. The mechanism that carries the argument is the dynamic focus state, a scalar score per entity updated by a fixed rule p_i(o) = clip(γ p_{i-1}(o) + α·φ⁺_i(o) − β·φ⁻_i(o)), where φ⁺ aggregates visibility, reappearance, state/location change, event participation, and interaction relevance, and φ⁻ aggregates absence and static background behavior. This focus state induces per-segment writing levels that tell the write

What would settle it

Build a ground-truth state timeline for a set of streaming videos annotated with scoreboard changes, object locations, and action transitions, then compare FOLIO's written entity records against that timeline. If the records frequently omit score changes or fine-grained action transitions (as the paper's own failure analysis suggests), yet the accuracy figures still hold, then the accuracy gains would not come from the memory records; alternatively, if record fidelity is high but accuracy does not track it, the focus-guided writing claim would need revision. A more direct test: replace FOLIO's

Watch

Extended reading notes

Core claim

FOLIO shows that an online video memory system can get better streaming question-answering accuracy while writing less, if it writes around entities rather than around time. Its long-term semantic memory is an entity-centered ledger: each observed person, object, or text region gets a stable identity slot with aliases, attributes, a location chain, an action chain, relations, and links to cached keyframes. A focus state, updated per segment from visibility, reappearance, state change, and interaction relevance, assigns each entity a writing level (focus, support, context, or drop) so that important records get fine detail and surrounding context stays compact. At query time, lightweight retr

Load-bearing premise

The writer vision-language model, given a few selected keyframes per 8-second segment and a writing-budget instruction, produces entity and action records faithful enough that later queries can be answered from those records.

Editorial extensions

If this is right

  • Long-horizon backward questions, such as where an object was minutes earlier or whether an event ever happened, can be answered from an entity ledger built online, reducing the need to keep raw frames or dense event summaries.
  • Multi-turn dialogue can steer memory construction: entities mentioned in earlier questions get a focus boost, so later turns about the same targets are grounded in more detailed records.
  • Focus-guided writing cuts the cost of maintaining streaming memory by around a third, without sacrificing accuracy, meaning memory size need not grow linearly with stream length when only a few entities matter.
  • Structured entity memory gives a direct way to detect hallucinations: a question about an entity or event that never appears can be answered 'unable to answer' from presence evidence in the ledger.
  • The method is training-free and works with the same VLM used for writing, retrieval expansion, and answering, so it can be applied to new backbones without fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If writer fidelity is the main bottleneck, then a better writer or a schema with domain-specific fields (scores, possession, fine-grained gestures) would amplify FOLIO's gains; the paper's own failure analysis points in this direction.
  • The entity-ledger idea should transfer beyond video to any streaming perception task where questions reference persistent objects, such as embodied robots or AR assistants, where a compact object history could replace raw sensor logs.
  • A testable extension is making the writing budget depend on predicted query difficulty or user history rather than a fixed per-segment budget, potentially preserving the cost savings while capturing rare but important state changes.
  • The ~32% cost reduction was measured on a diagnostic split; scaling to longer streams with many concurrent entities may change the write-detail tradeoff, so the entity-merge step's behavior on streams with hundreds of objects is worth separate study.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. FOLIO is a training-free streaming-video memory system that writes, at segment granularity, an entity-centered semantic memory guided by a dynamic focus state, alongside a short-term visual buffer and a visual-evidence cache. At query time it uses lightweight hybrid retrieval over the structured ledger, with VLM-assisted semantic expansion when direct matching fails. The paper reports state-of-the-art accuracy on OVO-Bench (82.0 Perception / 69.1 Backward with Qwen3-VL-8B) and StreamingBench (74.5 overall), with reduced writer cost relative to fixed-uniform writing. The main design claims are supported by component ablations, cost analyses, qualitative cases, and verbatim prompt templates.

Significance. If confirmed, FOLIO would be a useful instantiation of entity-centric, focus-guided memory for online video QA, and the paper's combination of structured memory, explicit focus-state updates, and recoverable keyframe evidence is well matched to backward-tracing and multi-turn questions. The manuscript is unusually transparent in several respects: the writer, expansion, and answer prompts are reproduced verbatim; the entity-merging and relevance-scoring equations are given; and the failure taxonomy separates missing memory from retrieval failures. The central quantitative claim, however, is weakened by the treatment of the HLD subtask, and the marginal contribution of the long-term semantic memory over the recent visual window is modest once HLD is discounted. The architectural ideas remain worth publishing after a corrected and more carefully controlled comparison.

major comments (4)
  1. [§4.2, Table 1, Appendix K.3] The headline Backward claim is dominated by HLD. Qwen3-VL-8B+FOLIO's Backward average of 69.1 includes HLD=84.6; the paper itself states that HLD 'benefits from conservative answer calibration' and should be reported separately, yet Table 1 includes it in the average and no HLD-excluded row is given. The Mode A answer prompt in Appendix K.3 explicitly instructs the model to select 'Unable to answer' unless memory/frames contain DIRECT, EXPLICIT evidence, and bans speculative wording. Since HLD questions are deliberately irrelevant and the correct answer is typically 'Unable to answer,' the 84.6 HLD score largely measures prompt calibration rather than memory quality. Excluding HLD, FOLIO's Backward average is (59.0+63.6)/2=61.3 versus OASIS's (62.0+60.1)/2=61.05, a 0.25-point margin rather than the claimed +11.9. The state-of-the-art statement and the 'especially clear on Backward querie
  2. [§4.4, Table 5] The evidence ablation does not show a large standalone contribution of the long-term semantic memory on the OVO-Bench Backward split. On the 200-query split, short-term visual buffer alone (S) attains 54.5, S+O attains 55.5, and the full FOLIO policy attains 56.7; O alone is 42.6. Thus the structured memory adds roughly 1–2 points over the recent-frame baseline on this split, and the stronger claimed Backward advantage appears only in the HLD-heavy full-table comparison. Because the paper argues that entity-centered memory is the key differentiator over event memories such as OASIS, the authors should report Backward accuracy by subtask with HLD excluded, and should state how much of the remaining gain comes from the structured ledger versus the answer prompt and the recent window.
  3. [§3.2, §4.3, Fig. 3] The fixed-hyperparameter claim is not verifiable as written. Equation (1) and Eqs. (5)–(6) introduce coefficients gamma, alpha, beta, retrieval weights w_T,w_H,w_Y,w_V,w_B, entity-matching weights lambda_*, and thresholds tau_m, all described as fixed method hyperparameters, but their values are not given anywhere in the main text or appendices. In addition, the default segment length L=8s is selected using a 200-video StreamingBench diagnostic subset (Fig. 3), and the cost tables (Table 4) use the same 8s setting. Without numerical values and with segment length chosen on an evaluation subset, the claims 'no per-dataset tuning' and the reported cost/accuracy tradeoff are difficult to reproduce and are at risk of diagnostic-subset overfitting. The authors should disclose the hyperparameter values and, if possible, report the sensitivity of the main tables to L.
  4. [§3.4, Appendix K.1, Table 11] The system's accuracy depends on the writer VLM producing structured entity/action records that are faithful enough for later queries, but writer fidelity is never measured against ground truth. Table 11 acknowledges that a substantial share of OVO-Bench errors (36%) are 'memory missing' cases where the writer fails to capture fine-grained states, OCR/score changes, or action transitions, and the qualitative cases in Appendix F show similar failures. Since the cost/accuracy advantage of FOLIO is predicated on focused write detail, the paper should either provide a direct fidelity evaluation of writer records (e.g., precision/recall of entity states against annotated keyframes) or explicitly bound the effect of writer misses on the reported accuracy. At minimum, the failure taxonomy should be accompanied by error counts per OVO-Bench subtask, especially EPM and ASI.
minor comments (5)
  1. [§4.2] The sentence 'HLD ... should be reported separately' is not operationalized: Table 1 does not include a separate HLD-excluded Backward average. Please add a row or column for Backward minus HLD.
  2. [Abstract / Table 1] 'State-of-the-art' is a strong claim given that Table 1 compares only OASIS among online memory baselines on OVO-Bench, and the comparison excludes the FAR subtasks. Please qualify the claim or add more baselines.
  3. [§4.4 / Appendix G] Main text refers to 'Appendix 11' for the failure taxonomy; the actual location is Appendix G / Table 11. Please fix the cross-reference.
  4. [Appendix K.3] Mode A's banned-word list ('most likely', 'suggests', 'implies', 'could be', 'probably') is a strong prior toward abstention. This should be acknowledged as a design choice that can help HLD while potentially hurting other Backward tasks, and its effect should be measured.
  5. [Table 4] Writer latency of 5.88–7.68s per 8s chunk means the writer consumes most of the segment budget. The paper mentions batching as future work, but a sentence connecting this to real-time compliance (or lack thereof) would clarify the streaming claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's accuracy claims are empirical benchmark measurements, and its design equations do not encode the reported outcomes.

full rationale

FOLIO's central claims are validated by direct benchmark measurements against external baselines (OVO-Bench, StreamingBench, StreamBench), not derived from fitted equations or from self-citation. The focus-state recurrence (Eq. 1), writing-level rule (Eq. 10), relevance score (Eq. 5), and entity matcher (Eq. 6) are algorithmic definitions that do not assume the target accuracies. No self-citations appear in the reference list; load-bearing prior work (OASIS [29], Think-While-Watching [50], OVO-Bench [40], StreamingBench [31]) is external. The paper explicitly flags the main interpretive caveat in Section 4.2: "HLD also benefits from conservative answer calibration, so we report it separately and do not use it as the sole evidence for memory quality." Appendix K.3's Mode A prompt indeed instructs the model to select "Unable to answer" absent direct evidence, which affects the HLD component of the Backward average; this is a benchmark-attribution and reporting concern, not a circular derivation, because the paper discloses the prompt dependence and the rest of the Backward score is empirically measured. Similarly, choosing the 8-s segment length on a 200-video diagnostic subset (Fig. 3) and using undisclosed fixed hyperparameters are tuning choices, not fitted parameters renamed as predictions. The failure taxonomy (Table 11) and ablations (Fig. 4) honestly admit writer-fidelity limits and the importance of the recent visual window; these are correctness risks, not evidence that the results reduce to their inputs by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or metaphysical entities are postulated; the object ledger, focus state, and evidence cache are algorithmic structures with no external falsifiable predictions. The central claim rests instead on hand-set hyperparameters, VLM reliability, and rule-based identity matching.

free parameters (5)
  • Segment length L (default 8 s) = 8 s
    Selected from chunk-length sensitivity on a 200-video StreamingBench diagnostic subset (Fig. 3); not derived from theory.
  • Focus update coefficients gamma, alpha, beta = not disclosed
    Fixed hyperparameters in Eq. 1/8 controlling focus persistence and positive/negative feature weights; no values or sensitivity analysis given.
  • Retrieval weights w_T, w_H, w_Y, w_V, w_B and relevance threshold = not disclosed
    Appear symbolically in Eq. 5; no numeric values or threshold given, so retrieval ranking is not reproducible from the paper alone.
  • Entity matching weights lambda_name/alias/head/cat/sub and threshold tau_m = not disclosed
    Eq. 6; no numeric values supplied; identity-merging behavior cannot be replicated exactly.
  • Writer budget top_k_objects and top_m_events = not disclosed
    Used in writer prompt (Appendix K.1); no values stated.
assumptions (4)
  • domain assumption Writer VLM produces faithful structured records from sparse keyframes.
    Central memory quality depends on the Qwen VLM obeying the budget prompt and capturing fine-grained state/OCR/actions; the paper's own failure taxonomy (Table 11) shows this often fails.
  • domain assumption Rule-based identity matching preserves entity identity across segments.
    Eqs. 6-7; if merges are wrong, location/action chains corrupt. No quantitative identity-error evaluation is provided.
  • domain assumption OVO-Bench RTV+BT and StreamingBench are valid proxies for streaming video understanding.
    Used as ground truth; FAR subtasks of OVO-Bench are excluded, so the 'OVO-Bench' claim is scoped to 9 of 12 subtasks.
  • ad hoc to paper Fixed hyperparameters generalize across datasets without per-dataset tuning.
    Authors state coefficients are fixed method hyperparameters, but no cross-dataset sensitivity analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FOLIO: Focused Semantic Memory for Streaming Video Understanding." pith.science (2026). https://pith.science/paper/EHKUP24E

@misc{pith2026260713298,
  author       = {Pith},
  title        = {Pith review of: FOLIO: Focused Semantic Memory for Streaming Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EHKUP24E}},
  note         = {Machine review of arXiv:2607.13298}
}
read the original abstract

In online streaming video understanding, a video stream continues to arrive and queries may be issued at any time. Because streaming frames grow without bound, the system must continuously compress and retain information from the observed video prefix while future frames and future queries remain unknown. The core challenge is deciding what information to retain and how to organize the maintained history: as this history grows with the stream, memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes. To address this challenge, we introduce FOLIO, a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context compact. As the stream arrives, FOLIO updates memory at the segment level, guided by a dynamic focus state, combining a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual-evidence cache. At query time, lightweight hybrid retrieval combines direct matching over the structured memory with semantic query expansion. FOLIO achieves state-of-the-art performance, reaching 82.0/69.1 Perception/Backward accuracy on OVO-Bench with Qwen3-VL-8B and 74.5 overall accuracy on StreamingBench, while substantially reducing the cost of maintaining streaming memory by reserving detailed records for focused entities and storing surrounding context compactly.

Figures

Figures reproduced from arXiv: 2607.13298 by the authors.

Figure 1
Figure 1. FOLIO overview. A training-free focused semantic memory system for streaming video understanding. (1) Online segment-by￾segment keyframe selection provides visual evidence for focus-guided memory writing. (2) A hybrid memory combines a short-term visual buffer with a long-term semantic memory linked to a visual-evidence cache. (3) At query time, FOLIO uses lightweight hybrid retrieval to link the query to relevant m… view at source ↗
Figure 2
Figure 2. FOLIO overview. A streaming cooking video runs across the top with an illustrative query at tq=60 s. (a) For each segment Ci, a writer VLM produces detailed/compact records under focus-guided writing levels Λi=BUDGET(Fi−1) and updates the hybrid memory Mt=(St, Ot, Bt); UPDATEFOCUS refreshes Fi from signals ϕ ± i . (b) A query is parsed to zq, linked to a ranked entity subset Oq (with a SEMLINK VLM fallback), assembl… view at source ↗
Figure 3
Figure 3. Chunk-length sensitivity using Qwen3-VL-8B on the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Component ablation using Qwen3-VL-8B on the 200- [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Interactive demo browser for inspecting video, queries, merged memory, and chunk-level memory in one view. [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

109 extracted references · 21 linked inside Pith

  1. [1]

    Goldfish: Vision- language understanding of arbitrarily long videos

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Mingchen Zhuge, Jian Ding, Deyao Zhu, Jürgen Schmidhuber, and Mohamed Elhoseiny. Goldfish: Vision- language understanding of arbitrarily long videos. InEu- ropean Conference on Computer Vision, pages 251–267. Springer, 2024. 3, 15, 16

    2024

  2. [2]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025. 3, 15

    arXiv 2025

  3. [3]

    Qwen2.5- vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report, ...

    2025

  4. [4]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakr- ishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnik...

    arXiv 2023

  5. [5]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18407–18418, 2024. 1, 3, 16

    2024

  6. [6]

    Livecc: Learning video llm with streaming speech transcription at scale

    Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livecc: Learning video llm with streaming speech transcription at scale. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 29083– 29095, 2025. 3, 16

    2025

  7. [7]

    Stream- ingtom: Streaming token compression for efficient video understanding.arXiv preprint arXiv:2510.18269, 2025

    Xueyi Chen, Keda Tao, Kele Shao, and Huan Wang. Stream- ingtom: Streaming token compression for efficient video understanding.arXiv preprint arXiv:2510.18269, 2025. 3, 16

    arXiv 2025

  8. [8]

    Streamkv: Streaming video question- answering with segment-based kv cache retrieval and com- pression

    Yilong Chen, Xiang Bai, Zhibin Wang, Chengyu Bai, Yuhan Dai, and Ming Lu. Streamkv: Streaming video question- answering with segment-based kv cache retrieval and com- pression. InProceedings of the AAAI Conference on Artificial Intelligence, pages 3120–3128, 2026. 1

    2026

Show all 109 references
  1. [9]

    Streaming video question-answering with in-context video kv-cache retrieval

    Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. Streaming video question-answering with in-context video kv-cache retrieval. InInternational Conference on Learning Representations, pages 42115–42127, 2025. 1, 3, 16

    2025

  2. [10]

    Streammind: Un- locking full frame rate streaming video dialogue through event-gated cognition

    Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Qianxi Zhang, Donglin Bai, Zhibo Chen, and Ting Cao. Streammind: Un- locking full frame rate streaming video dialogue through event-gated cognition. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 13...

    2025

  3. [11]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the IEEE/CVF conference on ...

    2025

  4. [12]

    Vispeak: Visual instruction feedback in streaming videos

    Shenghao Fu, Qize Yang, Yuan-Ming Li, Yi-Xing Peng, Kun- Yu Lin, Xihan Wei, Jian-Fang Hu, Xiaohua Xie, and Wei-Shi Zheng. Vispeak: Visual instruction feedback in streaming videos. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 21778–21788, 2...

    2025

  5. [13]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Ham- burger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ra- makrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Me...

    2022

  6. [14]

    Video streaming thinking: Videollms can watch and think simulta- neously.arXiv preprint arXiv:2603.12262, 2026

    Yiran Guan, Liang Yin, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, and Xiang Bai. Video streaming thinking: Videollms can watch and think simulta- neously.arXiv preprint arXiv:2603.12262, 2026. 3, 16

    2026 arXiv

  7. [15]

    Event-vstream: Event-driven real- time understanding for long video streams.arXiv preprint arXiv:2601.15655, 2026

    Zhenghui Guo, Yuanbin Man, Junyuan Sheng, Bowen Lin, Ahmed Ahmed, Bo Jiang, Boyuan Zhang, Miao Yin, Sian Jin, Omprakash Gnawal, et al. Event-vstream: Event-driven real- time understanding for long video streams.arXiv preprint arXiv:2601.15655, 2026. 3, 16

    2026

  8. [16]

    Wat: Online video understanding needs watching before thinking.arXiv preprint arXiv:2603.13412, 2026

    Zifan Han, Hongbo Sun, Jinglin Xu, Canhui Tang, Yulong Lei, Xuchong Zhang, Hongbin Sun, Zhongjiang He, and Hao Sun. Wat: Online video understanding needs watching before thinking.arXiv preprint arXiv:2603.13412, 2026. 3, 16

    2026

  9. [17]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. InProceedings of the IEEE/CVF conference on computer vision and pattern recog- n...

    2024

  10. [18]

    Online video understanding: Ovbench and videochat- online

    Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: Ovbench and videochat- online. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 3328–3338, 2025. 1, 3, 16

    2025

  11. [19]

    Online video understanding: Ovbench and videochat- online

    Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: Ovbench and videochat- online. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 3328–3338, 2025. 3, 16

    2025

  12. [20]

    Egospeak: learning when to speak for egocentric conver- sational agents in the wild

    Junhyeok Kim, Min Soo Kim, Jiwan Chung, Jungbin Cho, Jisoo Kim, Sungwoong Kim, Gyeongbo Sim, and Youngjae Yu. Egospeak: learning when to speak for egocentric conver- sational agents in the wild. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 2990–3005,

    2025

  13. [21]

    Infinipot-v: Memory-constrained kv cache compres- sion for streaming video understanding.Advances in Neural Information Processing Systems, 38:138983–139013, 2026

    Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. Infinipot-v: Memory-constrained kv cache compres- sion for streaming video understanding.Advances in Neural Information Processing Systems, 38:138983–139013, 2026. 1, 3, 16

    2026

  14. [22]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention.arXiv preprint arXiv:2309.06180, 2023. 6

    2023 arXiv

  15. [23]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195– ...

    2024

  16. [24]

    Videochat: Chat-centric video understanding.Science China Information Sciences, 68(10):200102, 2025

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.Science China Information Sciences, 68(10):200102, 2025. 3, 15

    2025

  17. [25]

    Freshmem: Brain-inspired frequency-space hybrid memory for streaming video understanding.arXiv preprint arXiv:2602.01683, 2026

    Kangcong Li, Peng Ye, Lin Zhang, Chao Wang, Huafeng Qin, and Tao Chen. Freshmem: Brain-inspired frequency-space hybrid memory for streaming video understanding.arXiv preprint arXiv:2602.01683, 2026. 1, 3, 16

    2026

  18. [26]

    Lion-fs: Fast & slow video-language thinker as online video assistant

    Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. Lion-fs: Fast & slow video-language thinker as online video assistant. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3240–3251,

  19. [27]

    Llama-vid: An im- age is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An im- age is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pages 323–340. Springer,

  20. [28]

    From verbatim to gist: Distilling pyramidal multimodal memory via semantic information bottleneck for long-horizon video agents.arXiv preprint arXiv:2603.01455, 2026

    Niu Lian, Yuting Wang, Hanshu Yao, Jinpeng Wang, Bin Chen, Yaowei Wang, Min Zhang, and Shu-Tao Xia. From verbatim to gist: Distilling pyramidal multimodal memory via semantic information bottleneck for long-horizon video agents.arXiv preprint arXiv:2603.01455, 2026. 16

    2026 arXiv

  21. [29]

    Oasis: On-demand hierarchical event memory for streaming video reasoning.arXiv preprint arXiv:2604.17052, 2026

    Zhijia Liang, Jiaming Li, Weikai Chen, Yanhao Zhang, Hao- nan Lu, and Guanbin Li. Oasis: On-demand hierarchical event memory for streaming video reasoning.arXiv preprint arXiv:2604.17052, 2026. 1, 3, 6, 16, 20

    2026 arXiv

  22. [30]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 5971–5984, 2024. 1, 3, 15

    2024

  23. [31]

    Streamingbench: Assessing the gap for mllms to achieve streaming video understanding

    Junming Lin, Zheng Fang, Chi Chen, Haoxuan Cheng, Zi- hao Wan, Fuwen Luo, Ziyue Wang, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. InICASSP 2026- 2026 IEEE International Conference on Acoustics, Speec...

    2026

  24. [32]

    Speak while watch- ing: Unleashing true real-time video understanding capabil- ity of multimodal large language models.arXiv preprint arXiv:2601.06843, 2026

    Junyan Lin, Junlong Tong, Hao Wu, Jialiang Zhang, Jin- ming Liu, Xin Jin, and Xiaoyu Shen. Speak while watch- ing: Unleashing true real-time video understanding capabil- ity of multimodal large language models.arXiv preprint arXiv:2601.06843, 2026. 3, 16

    2026

  25. [33]

    Stream- chat: Chatting with streaming video.arXiv preprint arXiv:2412.08646, 2024

    Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, and Jose M Alvare. Stream- chat: Chatting with streaming video.arXiv preprint arXiv:2412.08646, 2024. 16

    2024 arXiv

  26. [34]

    Thinking in streaming video.arXiv preprint arXiv:2603.12938, 2026

    Zikang Liu, Longteng Guo, Handong Li, Ru Zhen, Xingjian He, Ruyi Ji, Xiaoming Ren, Yanhao Zhang, Haonan Lu, and Jing Liu. Thinking in streaming video.arXiv preprint arXiv:2603.12938, 2026. 3, 16

    2026

  27. [35]

    Vista: Scene-aware optimization for streaming video question answering under post-hoc queries

    Haocheng Lu, Nan Zhang, Wei Tao, Xiaoyang Qu, Guokuan Li, Jiguang Wan, and Jianzong Wang. Vista: Scene-aware optimization for streaming video question answering under post-hoc queries. InProceedings of the AAAI Conference on Artificial Intelligence, pages 7539–7547, 2026. 1, 3, 16

    2026

  28. [36]

    Aura: Always-on understanding and real-time assistance via video streams.arXiv preprint arXiv:2604.04184, 2026

    Xudong Lu, Yang Bo, Jinpeng Chen, Shuhan Li, Xintong Guo, Huankang Guan, Fang Liu, Dunyuan Xu, Peiwen Sun, Heyang Sun, et al. Aura: Always-on understanding and real-time assistance via video streams.arXiv preprint arXiv:2604.04184, 2026. 3, 16 10

    2026 arXiv

  29. [37]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension.Advances in Neural Information Processing Systems, 38:168008–168033, 2026

    Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, et al. Video-rag: Visually-aligned retrieval-augmented long video comprehension.Advances in Neural Information Processing Systems, 38:168008–168033, 2026. 3, 15, 16

    2026

  30. [38]

    Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 1, 3, 15

    2023

  31. [39]

    Livevlm: Efficient online video understanding via streaming-oriented kv cache and retrieval.arXiv preprint arXiv:2505.15269, 2025

    Zhenyu Ning, Guangda Liu, Qihao Jin, Chengwei Li, Wen- chao Ding, Minyi Guo, and Jieru Zhao. Livevlm: Efficient online video understanding via streaming-oriented kv cache and retrieval.arXiv preprint arXiv:2505.15269, 2025. 3, 16

    2025 arXiv

  32. [40]

    Ovo-bench: How far is your video-llms from real-world online video understanding? InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18902–18913, 2025

    Junbo Niu, Yifei Li, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. Ovo-bench: How far is your video-llms from real-world online video understanding? InProceedings of the Computer Vision and Pattern Recognition C...

    2025

  33. [41]

    Streaming long video un- derstanding with large language models.Advances in Neural Information Processing Systems, 37:119336–119360, 2024

    Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video un- derstanding with large language models.Advances in Neural Information Processing Systems, 37:119336–119360, 2024. 3, 15, 16

    2024

  34. [42]

    Video-xl-2: Towards very long-video under- standing through task-aware kv sparsification.arXiv preprint arXiv:2506.19225, 2025

    Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao, and Zheng Liu. Video-xl-2: Towards very long-video under- standing through task-aware kv sparsification.arXiv preprint arXiv:2506.19225, 2025. 15

    2025 arXiv

  35. [43]

    Longvu: Spa- tiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Bal- akrishnan Varadarajan, Florian Bordes, et al. Longvu: Spa- tiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 202...

    2024 arXiv

  36. [44]

    A simple baseline for streaming video understanding.arXiv preprint arXiv:2604.02317, 2026

    Yujiao Shen, Shulin Tian, Jingkang Yang, and Ziwei Liu. A simple baseline for streaming video understanding.arXiv preprint arXiv:2604.02317, 2026. 1, 3, 4, 16

    2026

  37. [45]

    Video- xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video- xl: Extra-long vision language model for hour-scale video understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 26160–26169, ...

    2025

  38. [46]

    Drivelm: Driving with graph visual question answering.arXiv preprint arXiv:2312.14150, 2023

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering.arXiv preprint arXiv:2312.14150, 2023. 1

    2023 arXiv

  39. [47]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  40. [48]

    Curvestream: Boosting streaming video understanding in mllms via curvature-aware hierarchical visual memory management.arXiv preprint arXiv:2603.19571, 2026

    Chao Wang, Xudong Tan, Jianjian Cao, Kangcong Li, and Tao Chen. Curvestream: Boosting streaming video understanding in mllms via curvature-aware hierarchical visual memory management.arXiv preprint arXiv:2603.19571, 2026. 1, 3, 16

    2026

  41. [49]

    Streambridge: Turning your offline video large language model into a proactive streaming assistant.Advances in Neural Information Processing Systems, 38:132332–132359,

    Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, and Ping Huang. Streambridge: Turning your offline video large language model into a proactive streaming assistant.Advances in Neural Information Processing Systems, 38:132332–132359,

  42. [50]

    Think while watching: Online streaming segment-level memory for multi-turn video rea- soning in multimodal large language models.arXiv preprint arXiv:2603.11896, 2026

    Lu Wang, Zhuoran Jin, Yupu Hao, Yubo Chen, Kang Liu, Yulong Ao, and Jun Zhao. Think while watching: Online streaming segment-level memory for multi-turn video rea- soning in multimodal large language models.arXiv preprint arXiv:2603.11896, 2026. 3, 6, 16

    2026

  43. [51]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 1, 3, 15

    2024 arXiv

  44. [52]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, et al. Lvbench: An extreme long video understanding benchmark. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22958–22967, 2025. 3, 15

    2025

  45. [53]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent. InEuropean Conference on Computer Vision, pages 58–76. Springer, 2024. 3, 15, 16

    2024

  46. [54]

    Internvideo2: Scaling foundation models for mul- timodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for mul- timodal video understanding. InEuropean conference on computer vision, pages 396–416. Springer, 2024. 1, 3, 15

    2024

  47. [55]

    Vide- ollm knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format.arXiv preprint arXiv:2411.17991, 1(3):5, 2024

    Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. Vide- ollm knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format.arXiv preprint arXiv:2411.17991, 1(3):5, 2024. 1, 3, 16

    2024

  48. [56]

    Acceler- ating streaming video large language models via hierarchical token compression.arXiv preprint arXiv:2512.00891, 2025

    Yiyu Wang, Xuyang Liu, Xiyan Gui, Xinying Lin, Boxue Yang, Chenfei Liao, Tailai Chen, and Linfeng Zhang. Acceler- ating streaming video large language models via hierarchical token compression.arXiv preprint arXiv:2512.00891, 2025. 1, 3, 16

    2025

  49. [57]

    Omnimmi: A comprehensive multi- modal interaction benchmark in streaming video contexts

    Yuxuan Wang, Yueqian Wang, Bo Chen, Tong Wu, Dongyan Zhao, and Zilong Zheng. Omnimmi: A comprehensive multi- modal interaction benchmark in streaming video contexts. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18925–18935, 2025. 3, 16

    2025

  50. [58]

    Episodic memory representation for long- form video understanding.arXiv preprint arXiv:2508.09486,

    Yun Wang, Long Zhang, Jingren Liu, Jiaqi Yan, Zhanjie Zhang, Jiahao Zheng, Xun Yang, Dapeng Wu, Xiangyu Chen, and Xuelong Li. Episodic memory representation for long- form video understanding.arXiv preprint arXiv:2508.09486,

  51. [59]

    11 Videotree: Adaptive tree-based video representation for llm reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. 11 Videotree: Adaptive tree-based video representation for llm reasoning on long videos. InProceedings of the Computer Vi- sion and Pattern Recognition Conference, pages ...

  52. [60]

    Eventmemagent: Hierarchical event-centric memory for online video understanding with adaptive tool use.arXiv preprint arXiv:2602.15329, 2026

    Siwei Wen, Zhangcheng Wang, Xingjian Zhang, Lei Huang, and Wenjun Wu. Eventmemagent: Hierarchical event-centric memory for online video understanding with adaptive tool use.arXiv preprint arXiv:2602.15329, 2026. 1, 3, 16

    2026

  53. [61]

    Videollm-mod: Efficient video-language streaming with mixture-of-depths vision computation.Ad- vances in Neural Information Processing Systems, 37:109922– 109947, 2024

    Shiwei Wu, Joya Chen, Kevin Qinghong Lin, Qimeng Wang, Yan Gao, Qianli Xu, Tong Xu, Yao Hu, Enhong Chen, and Mike Zheng Shou. Videollm-mod: Efficient video-language streaming with mixture-of-depths vision computation.Ad- vances in Neural Information Processing Systems, 37:1099...

    2024

  54. [62]

    Next-qa: Next phase of question-answering to explaining tem- poral actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining tem- poral actions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786,

  55. [63]

    Fluxmem: Adaptive hierarchical memory for streaming video understanding.arXiv preprint arXiv:2603.02096, 2026

    Yiweng Xie, Bo He, Junke Wang, Xiangyu Zheng, Ziyi Ye, and Zuxuan Wu. Fluxmem: Adaptive hierarchical memory for streaming video understanding.arXiv preprint arXiv:2603.02096, 2026. 1, 3, 16

    2026

  56. [64]

    Streaming video under- standing and multi-round interaction with memory-enhanced knowledge.arXiv preprint arXiv:2501.13468, 2025

    Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, and Huchuan Lu. Streaming video under- standing and multi-round interaction with memory-enhanced knowledge.arXiv preprint arXiv:2501.13468, 2025. 15, 20

    2025 arXiv

  57. [65]

    Streamingvlm: Real-time understanding for infinite video streams.arXiv preprint arXiv:2510.09608, 2025

    Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. Streamingvlm: Real-time understanding for infinite video streams.arXiv preprint arXiv:2510.09608, 2025. 3, 16

    2025 arXiv

  58. [66]

    RTV-bench: Benchmarking MLLM continuous per- ception, understanding and reasoning through real-time video

    ShuHang Xun, Sicheng Tao, Jungang Li, Yibo Shi, Zhixin Lin, Zhanhui Zhu, Yibo Yan, Hanqian Li, LingHao Zhang, Shikang Wang, Yixin Liu, Hanbo Zhang, Ying Ma, and Xum- ing Hu. RTV-bench: Benchmarking MLLM continuous per- ception, understanding and reasoning through real-time vid...

    2026

  59. [67]

    Streammem: Query-agnostic kv cache memory for streaming video understanding.arXiv preprint arXiv:2508.15717, 2025

    Yanlai Yang, Zhuokai Zhao, Satya Narayan Shukla, Aashu Singh, Shlok Kumar Mishra, Lizhu Zhang, and Mengye Ren. Streammem: Query-agnostic kv cache memory for streaming video understanding.arXiv preprint arXiv:2508.15717, 2025. 3, 16

    2025 arXiv

  60. [68]

    Svbench: A benchmark with temporal multi-turn dialogues for streaming video understanding.arXiv preprint arXiv:2502.10810, 2025

    Zhenyu Yang, Yuhang Hu, Zemin Du, Dizhan Xue, Sheng- sheng Qian, Jiahong Wu, Fan Yang, Weiming Dong, and Changsheng Xu. Svbench: A benchmark with temporal multi-turn dialogues for streaming video understanding.arXiv preprint arXiv:2502.10810, 2025. 1, 3, 16

    2025

  61. [70]

    Livestar: Live streaming assis- tant for real-world online video understanding.Advances in Neural Information Processing Systems, 38:31266–31304,

    Zhenyu Yang, Kairui Zhang, Yuhang Hu, Bing Wang, Sheng- sheng Qian, Bin Wen, Fan Yang, Tingting Gao, Weiming Dong, and Changsheng Xu. Livestar: Live streaming assis- tant for real-world online video understanding.Advances in Neural Information Processing Systems, 38:31266–31304,

  62. [71]

    Timechat-online: 80% visual tokens are naturally redundant in streaming videos

    Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, et al. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. InProceedings of the 33rd ACM International Conference on Multimedia, pag...

    2025

  63. [72]

    Worldmm: Dynamic multimodal memory agent for long video reasoning.arXiv preprint arXiv:2512.02425, 2025

    Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, and Sung Ju Hwang. Worldmm: Dynamic multimodal memory agent for long video reasoning.arXiv preprint arXiv:2512.02425, 2025. 16

    2025

  64. [73]

    Eyes wide open: Ego proactive video-llm for streaming video.Ad- vances in Neural Information Processing Systems, 38:13420– 13463, 2026

    Xueyang Yu, Cheng Shi, Yang Wang, and Sibei Yang. Eyes wide open: Ego proactive video-llm for streaming video.Ad- vances in Neural Information Processing Systems, 38:13420– 13463, 2026. 1, 3, 16

    2026

  65. [74]

    Streamforest: Efficient online video understand- ing with persistent event memory.Advances in Neural In- formation Processing Systems, 38:75804–75835, 2026

    Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang, Jiaxin Li, Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, et al. Streamforest: Efficient online video understand- ing with persistent event memory.Advances in Neural In- formation Processing Systems, 38:75804–75835, 20...

    2026

  66. [75]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. InProceedings of the 2023 conference on empir- ical methods in natural language processing: system demon- strations, pages 543–553, 2023. 3, 15

    2023

  67. [76]

    Hermes: Kv cache as hierarchical memory for efficient streaming video understanding.arXiv preprint arXiv:2601.14724, 2026

    Haowei Zhang, Shudong Yang, Jinlan Fu, See-Kiong Ng, and Xipeng Qiu. Hermes: Kv cache as hierarchical memory for efficient streaming video understanding.arXiv preprint arXiv:2601.14724, 2026. 3, 16

    2026 arXiv

  68. [77]

    Think-as-you-see: Stream- ing chain-of-thought reasoning for large vision-language mod- els.arXiv preprint arXiv:2603.02872, 2026

    Jialiang Zhang, Junlong Tong, Junyan Lin, Hao Wu, Yirong Sun, Yunpu Ma, and Xiaoyu Shen. Think-as-you-see: Stream- ing chain-of-thought reasoning for large vision-language mod- els.arXiv preprint arXiv:2603.02872, 2026. 3, 16

    2026

  69. [78]

    Querystream: Advancing streaming video understanding with query-aware pruning and proactive response

    Kairui Zhang, Zhenyu Yang, Bing Wang, Shengsheng Qian, and Changsheng Xu. Querystream: Advancing streaming video understanding with query-aware pruning and proactive response. InThe Fourteenth International Conference on Learning Representations, 2026. 1, 3, 16

    2026

  70. [79]

    Internlm-xcomposer2

    Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. Internlm-xcomposer2. 5-omnilive: A comprehen- sive multimodal system for long-term streaming video and audio interactions.arXiv preprint arXiv:2412.09596...

    2024 arXiv

  71. [80]

    Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024. 1, 3, 15

    2024 arXiv

  72. [81]

    Llava-video: Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024. 1, 3, 15 12

    2024 arXiv

  73. [82]

    Proactive assistant dialogue generation from streaming egocentric videos

    Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, and Seungwhan Moon. Proactive assistant dialogue generation from streaming egocentric videos. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Proce...

    2025

  74. [83]

    Proactive assistant dialogue generation from streaming egocentric videos

    Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, and Seungwhan Moon. Proactive assistant dialogue generation from streaming egocentric videos. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Proce...

    2025

  75. [84]

    Hierarchical event memory for accurate and low- latency online video temporal grounding

    Minghang Zheng, Yuxin Peng, Benyuan Sun, Yi Yang, and Yang Liu. Hierarchical event memory for accurate and low- latency online video temporal grounding. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 21589–21599, 2025. 3, 16

    2025

  76. [85]

    recent” from “earlier

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2(5): 6, 2024. 1, 3, 15 13 A. Supplementary Overview Th...

    2024 arXiv

  77. [86]

    DETAILED objects: if visible, write structured state with location, holder, state, relation, interaction, state_change, evidence_summary, and evidence_frames

  78. [87]

    COMPACT objects: if visible, write one compact note with location/state/relation and evidence_frames

  79. [88]

    Put them in compact_objects unless they are central to the action, then put them in detailed_objects

    Always discover newly visible manipulated objects, tools, containers, appliances, cooking surfaces, sinks, screens/text, and unusual objects. Put them in compact_objects unless they are central to the action, then put them in detailed_objects

  80. [89]

    Do not ignore new or rare objects just because they are not listed in the budget

  81. [90]

    time": "{start_time:.1f}-{end_time:.1f}s

    Do not duplicate the same physical object across detailed_objects and compact_objects. Return ONLY valid JSON: { "time": "{start_time:.1f}-{end_time:.1f}s", "detailed_objects": [ { "name": "object name", "category": "category", "attributes": ["attr1"], "location": "where it is...

  82. [91]

    Only list visible or partially visible objects

  83. [92]

    Maximum {top_k_objects} total objects across both object lists and {top_m_events} events

  84. [93]

    Prefer detailed budget for repeated or actively manipulated focus objects

  85. [94]

    Preserve compact notes for tools, containers, appliances, cooking surfaces, sinks, screens/text, and newly appearing objects even when they are not the action center

  86. [95]

    avid reader

    evidence_frames can be approximate local frame indices inside this segment; use an empty list if you cannot localize the evidence. K.2. Semantic Query Expansion Prompt Invoked only when direct structured-memory retrieval returns an empty candidate set, typically on abstract or...

  87. [96]

    avid reader

    Identify which tracked objects/actions are CONCEPTUALLY related to the question. - Example: "avid reader" -> bookshelf, books, magazine, reading lamp - Example: "musician" -> piano, guitar, drum - Example: "what activity is happening" -> look at action list

  88. [97]

    Return their object_ids (3-7 most relevant)

  89. [98]

    relevant_object_ids

    Also explain briefly which option seems most supported. ## OUTPUT (JSON only, no other text) { "relevant_object_ids": ["obj_0001", "obj_0002", "obj_0005"], "reasoning": "Brief explanation of why these objects relate to the question concept.", "suggested_option": "A/B/C/D" } 26...

  90. [99]

    The RECENT FRAMES shown above (current visual state near question time)

  91. [100]

    ## (!) CONCEPT QUESTION -- SPECIAL HANDLING REQUIRED

    The CONSOLIDATED MEMORY below (chronological history from earlier in the video) The memory is built from object observations across the video, organized as time-ordered location chains and action chains. MODE SELECTION (auto-detected from memory): === MODE A -- STRICT FACTUAL ...

  92. [101]

    Recent frames inform CURRENT state only

    Memory is the PRIMARY source for past events. Recent frames inform CURRENT state only

  93. [102]

    For multiple choice, select one option label (A, B, C, or D)

  94. [103]

    Unable to answer

    If "Unable to answer", "cannot determine", "not visible", or similar is among the options AND memory/frames do NOT contain DIRECT, EXPLICIT evidence, you MUST select it

  95. [104]

    Unable to answer

    Avoid speculation: prefer "Unable to answer" (when available) over guessing

  96. [105]

    most likely

    BANNED words: "most likely", "suggests", "implies", "could be", "commonly", "typically", "probably". If used, answer is likely "Unable to answer"

  97. [106]

    is a reader

    Mere presence of an object does NOT confirm its location, action, or relation. === MODE B -- CONCEPT REASONING (only when memory has CONCEPT QUESTION header) === This mode is enabled by the rule-based retriever failing entirely on this question (suggesting it asks an abstract ...

  98. [107]

    INDIRECT evidence IS valid: bookshelf -> reader; piano -> musician; tools -> handyman

  99. [108]

    You do NOT need direct visual confirmation of the activity

  100. [109]

    suggests/implies

    The BANNED-words rule of Mode A does NOT apply here -- you may use "suggests/implies"

  101. [110]

    TARGET NOT FOUND

    Trust the LLM-suggested option (if any) unless contradicted. === COMMON === Trust the QUESTION CONTEXT and OPTION CHECK signals at the top of memory when present. Return ONLY valid JSON. ## QUESTION {question} ## OPTIONS {options_text} ## CONSOLIDATED MEMORY {memory_text} ## D...

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.