Pith. sign in

REVIEW 3 major objections 77 references

Proactive egocentric assistants must decide whether and when to intervene by reasoning over accumulated history, not by reacting to every detected event.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 04:59 UTC pith:TOZ3G7SO

load-bearing objection First real intervene-or-silent benchmark for multi-day egocentric video, plus a clean training-free memory agent; absolute numbers are modest and labels lean on Gemini, but the problem framing and ablations are solid. the 3 major comments →

arxiv 2607.11523 v1 pith:TOZ3G7SO submitted 2026-07-13 cs.CV cs.AI

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Gong Sitong , Tianyu Yan , Caixin Kang , Bo Zheng , Xiang Ruan , Huchuan Lu , Kaipeng Zhang , Yoichi Sato
show 1 more author
Yifei Huang
This is my paper
classification cs.CV cs.AI
keywords egocentric visionproactive assistancestreaming video understandingmemory-augmented agentretrieval-augmented reasoningknowledge graphtemporal memory horizonsVideo-LLM
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most video assistants either wait for a user question or fire a reply whenever a predefined event appears. This paper argues that continuous first-person video enables a third mode: the system itself must judge, from the user's history, current activity, and likely needs, whether speaking up would actually help. To make that claim testable, the authors release EgoServe, a benchmark of more than three thousand service instances spanning four time scales—from instant safety alerts to multi-day habit coaching—across ten service types. They also present EgoMemo, a training-free agent that keeps multi-scale text summaries, a growing knowledge graph, and visual embedding archives, then retrieves from all three before deciding whether to intervene and what to say. On EgoServe the agent roughly doubles the overall detection score of strong proprietary video models, and it stays competitive on existing online and offline egocentric benchmarks. The work therefore frames proactive assistance as a memory-conditioned decision problem and supplies both the test bed and a concrete baseline for attacking it.

Core claim

Proactive assistance in continuous egocentric video is a joint decision-and-generation task: at each timestep the agent must output a binary intervention decision together with a grounded response only when intervention is warranted, and that decision requires retrieval over multi-horizon memory rather than local event detection alone. EgoMemo realizes this with three complementary streaming memories and retrieval-augmented reasoning, establishing the strongest reported baselines on the new EgoServe benchmark while remaining competitive on five existing egocentric tasks.

What carries the argument

EgoMemo: a training-free, memory-augmented agent that incrementally builds multi-scale temporal summaries (clip / activity / session), an evolving semantic knowledge graph, and a visual embedding archive, then runs parallel temporal, graph, and visual retrieval followed by caption reconstruction before the reasoner outputs the intervention decision and response.

Load-bearing premise

The semi-automated labels (foundation-model generation from existing human notes, then dual-annotator consensus) correctly mark the moments when assistance is actually warranted and the service type the user would welcome.

What would settle it

Run EgoMemo (or a stronger successor) on a held-out multi-day egocentric stream whose intervention points were marked by independent human wearers in real time, then measure whether category-matched temporal F1 and user-rated helpfulness remain above the paper's reported EgoServe baselines; a large drop would falsify the claim that the current annotations and memory design capture genuine proactivity.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmarks that only score answer quality on pre-segmented clips will miss the core skill of deciding when to stay silent.
  • Streaming assistants need hierarchical text memory plus graph and visual stores if they are to handle multi-day habit and routine services.
  • Caption reconstruction after retrieval is necessary; raw retrieved snippets alone collapse episodic and long-term performance.
  • The same streaming-first memory stack can be reused, without architectural change, for offline long-form egocentric QA.
  • Absolute F1 scores remain low, so EgoServe will continue to expose failures in timing, category choice, and grounding for future systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If intervention timing is later trained from real wearer preference rather than annotated windows, the binary decision head may become the dominant source of gain over pure retrieval quality.
  • The same multi-horizon taxonomy could stress-test non-egocentric wearable streams (smart glasses audio, physiological sensors) once analogous service labels exist.
  • Name-based entity resolution is likely to be the first component to break under crowded multi-person scenes; visual-grounded entity linking would be a natural next stress test.
  • Low absolute F1 even for the best system implies that deployment safety may require an explicit “confidence-to-interrupt” threshold learned separately from response generation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper reframes proactive egocentric assistance as a context-dependent decision problem over continuous first-person video, contrasting reactive and semi-proactive paradigms. It introduces EgoServe, a benchmark of ~3.4k service instances across 4 temporal memory horizons and 10 service categories built from EgoLife, HoloAssist, and CaptainCook4D, and EgoMemo, a training-free agent that maintains multi-scale temporal summaries, an evolving knowledge graph, and a visual embedding archive, then performs retrieval-augmented reasoning to decide whether to intervene and what to say. On EgoServe, EgoMemo reports overall F1 8.0 versus 4.7 for GPT-5-mini and 3.5 for Qwen3-VL-Plus, with the largest gains on long-term categories; ablations isolate multi-scale memory, caption reconstruction, and the three retrieval pathways. The same architecture remains competitive or better on ESTP-Bench, OVO-Bench, EgoSchema, EgoTaskQA, and QAEgo4D without architectural change.

Significance. If the evaluation is taken as valid, the work is a clear contribution: it names and operationalizes a third paradigm (proactive intervention as an explicit decision over accumulated context), ships the first multi-horizon proactive egocentric benchmark with public code, and shows that a training-free, streaming memory stack can move the needle where plain VLMs score near zero on long-horizon services. Clean component ablations, dual-mode (proactive/reactive) unification, and strong offline transfer (e.g., EgoSchema 74.8) are real strengths. The absolute F1 remains low and the ground-truth construction is semi-automated, so the result is best read as establishing a diagnostic baseline and problem formulation rather than a solved capability; that is still valuable for the community if the benchmark validity concerns are addressed.

major comments (3)
  1. Sec. 3.4 and Appendix A (incl. Table 6): For Episodic and Long-Term services, Gemini is prompted to accumulate cross-interval cues and emit candidate triggers—the same style of multi-session linking EgoMemo is scored on. Manual verification checks temporal presence, category fit, and “practical utility” of the generated text under dual-annotator consensus (85.1% acceptance), not whether a real user would welcome interruption. This is load-bearing for Table 1’s headline gap (ML 4.9 / RO 11.8 vs. 0 for baselines). Please add (i) a human re-annotation or preference study on a stratified subsample of long-horizon instances, (ii) inter-annotator agreement on trigger windows and categories independent of the model-generated text, and (iii) an explicit discussion of residual annotation–model style alignment as a threat to validity.
  2. Sec. 3.1 task formulation vs. Sec. 3.5 evaluation protocol: The paper defines proactivity as deciding “whether, when, and how to intervene” and whether assistance would be welcome, but scoring only requires (1) temporal match within dataset-specific δ, (2) same service subcategory, and (3) LLM-judge rationality/effectiveness against reference responses (Fig. 6; Table 12). Silence quality, false-alarm cost, and user welcome-ness are not measured. Either narrow the claim language to “temporally and categorically correct, reference-aligned interventions,” or add metrics/experiments that evaluate over-triggering and human preference for intervene vs. remain silent.
  3. Table 1 and Sec. 5.1: The main comparison pits EgoMemo’s full memory stack against GPT-5-mini and Qwen3-VL-Plus. The manuscript does not fully specify what streaming context, history length, or retrieval (if any) those baselines receive at each timestep. Without a matched protocol (e.g., same clip captions + rolling window, or an oracle full-history baseline), the large long-horizon gap is partly expected by construction. Please document baseline input construction precisely and, if feasible, add a strong memory-equipped or full-history VLM baseline so the contribution of EgoMemo’s specific three-store design is isolated from “having memory at all.”

Circularity Check

0 steps flagged

No derivation circularity: EgoMemo’s F1 gains are empirical evaluations on a new benchmark plus five external ones, not predictions forced by fitted inputs or self-cited uniqueness.

full rationale

The paper’s load-bearing claim is empirical (EgoMemo overall F1 8.0 vs GPT-5-mini 4.7 on EgoServe; competitive on ESTP-Bench, OVO-Bench, EgoSchema, EgoTaskQA, QAEgo4D), not a first-principles derivation. EgoMemo is training-free: multi-scale summaries, knowledge graph, and visual archives are constructed from streaming captions and used for retrieval-augmented decisions; no parameter is fitted to a subset of EgoServe and then reported as a prediction of a related quantity. Self-citations to Vinci and EgoLife supply the reactive predecessor and multi-day video source, not the performance numbers or a uniqueness theorem that forbids alternatives. The semi-automated Gemini annotation pipeline (Sec. 3.4, App. A) is a validity risk for whether ground-truth triggers match real user welcome-ness, but that is evaluation bias, not circular reduction of a claimed derivation to its inputs. External-benchmark results without architectural change further keep the central claim independent of EgoServe’s construction. No equation or claimed prediction reduces by construction to a fitted constant or self-definitional identity.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central empirical claim rests on a newly defined service taxonomy, a semi-automated annotation process that treats foundation-model outputs as candidate ground truth, and a set of engineering hyperparameters (window sizes, δ tolerances, top-k, retrieval thresholds) that are chosen rather than derived. No new physical entities are postulated; the three memory stores are architectural components.

free parameters (3)
  • temporal tolerance δ = 60s / 25s / 10s
    Dataset-specific matching windows (60 s EgoLife, 25 s CaptainCook4D, 10 s HoloAssist) directly determine true-positive counts and therefore all F1 scores.
  • multi-scale summarization windows Wj, Wk = dataset-dependent (30s/5min/1h or 10s/1min/5min)
    Clip/activity/session windows (e.g., 30 s / 5 min / 1 h for EgoLife) control what context is available to the reasoner; chosen by hand per dataset duration.
  • top-k retrieval counts and similarity thresholds
    Control how much evidence reaches the reasoner; not derived from first principles.
axioms (3)
  • domain assumption Proactive assistance can be evaluated by matching predicted (timestamp, category) pairs to human-verified service instances within a fixed temporal tolerance and scoring response quality with an LLM judge.
    Sec. 3.5 evaluation protocol; the entire benchmark score rests on this operationalization.
  • ad hoc to paper The four temporal horizons and ten service subcategories exhaustively cover the space of useful proactive interventions in daily egocentric video.
    Sec. 3.3 taxonomy; invented for EgoServe and not independently validated against user studies of interruption desirability.
  • domain assumption Name-based entity resolution and 1-hop graph expansion suffice to link semantically related events across days.
    Sec. 4.1 knowledge-graph construction; acknowledged as a limitation for visually ambiguous entities.
invented entities (2)
  • EgoServe service taxonomy (4 horizons × 10 categories) no independent evidence
    purpose: Organizes ground-truth proactive interventions so that timing, type and response quality can be scored.
    Defined in Sec. 3.3; no external user study confirms that these categories match real user willingness to be interrupted.
  • EgoMemo three-memory store (multi-scale summaries + knowledge graph + visual archive) with VLM caption reconstruction no independent evidence
    purpose: Supplies the historical context that lets the reasoner decide intervene vs. silent.
    Architectural invention of Sec. 4; ablations show each component contributes, but the entity itself is not independently measured outside the paper’s own metrics.

pith-pipeline@v1.1.0-grok45 · 35541 in / 3075 out tokens · 37680 ms · 2026-07-14T04:59:52.753998+00:00 · methodology

0 comments
read the original abstract

When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at \href{https://sitonggong.github.io/EgoServe-page/}{Vinci2}.

Figures

Figures reproduced from arXiv: 2607.11523 by Bo Zheng, Caixin Kang, Gong Sitong, Huchuan Lu, Kaipeng Zhang, Tianyu Yan, Xiang Ruan, Yifei Huang, Yoichi Sato.

Figure 1
Figure 1. Figure 1: Three paradigms of egocentric assistants. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the EgoServe benchmark. (a) Annotation pipeline: human annotations from each source dataset are processed through category-specific prompts via a foundation model, followed by manual verification. (b) Response word frequency. (c) Video duration distribution. (d) Per-dataset service counts. (e) Service instance distribution across 10 subcategories and 4 temporal horizons. produce a binary interv… view at source ↗
Figure 3
Figure 3. Figure 3: Architecture of EgoMemo. (a) clip-level captions are incrementally or￾ganized into three-level temporal summaries (MC , MA, MS), an evolving knowledge graph G, and a visual embedding archive A. (b) Streaming retrieval-augmented reason￾ing: three parallel retrieval pathways (temporal, semantic, and visual) gather evidence, which is unified via VLM-based caption reconstruction before the reasoner produces an… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative examples of EgoMemo’s proactive assistance. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of the multi-scale temporal memory construction. Clip-level cap￾tions MC preserve fine-grained timestamped action descriptions, which are progres￾sively summarized into activity-level (MA) and session-level (MS) captions covering broader temporal spans. B.1 Details on Multi-Scale Temporal Memory Construction We provide additional details on the multi-scale temporal memory described in Sec. 4.1… view at source ↗
Figure 6
Figure 6. Figure 6: LLM Evaluation Prompts. Detection metrics. For each service sub-type s, we aggregate matched, pre￾dicted, and GT counts globally across all videos in the dataset and compute Precision, Recall, and F1: \text {P}_s = \frac {|\text {matched}_s|}{|\text {pred}_s|},\quad \text {R}_s = \frac {|\text {matched}_s|}{|\text {gt}_s|},\quad \text {F1}_s = \frac {2\,\text {P}_s\,\text {R}_s}{\text {P}_s + \text {R}_s}.… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative examples of EgoMemo’s proactive assistance. the value of long-term memory retrieval: on Day 5, the system detects the user checking their phone and retrieves evidence of repeated phone use during social gatherings across multiple earlier days, synthesizing this cross-day pattern into a coaching suggestion to reduce screen time. Notably, the streaming timeline shows that EgoMemo appropriately re… view at source ↗
Figure 8
Figure 8. Figure 8: Annotation examples of Instant proactive assistance. Annotation Content Annotation Content Time Window: DAY1 12:46:54-12:46:58 Observation: I was working on my laptop in the bedroom but got distracted by my phone and walked out to the kitchen, leaving the laptop open. Response: I noticed you headed to the kitchen, but your laptop is still open in the bedroom. Annotation Content Time Window: DAY1 00:08:26-0… view at source ↗
Figure 9
Figure 9. Figure 9: Annotation examples of Short-term proactive assistance [PITH_FULL_IMAGE:figures/full_fig_p029_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Annotation examples of Episodic proactive assistance [PITH_FULL_IMAGE:figures/full_fig_p030_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Annotation examples of Long-term proactive assistance [PITH_FULL_IMAGE:figures/full_fig_p031_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 17 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2309.16609 (2023) 12, 13, 14

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023) 12, 13, 14

    Pith/arXiv arXiv 2023

  2. [2]

    arXiv preprint arXiv:2511.21631 (2025) 11

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025) 11

    Pith/arXiv arXiv 2025

  3. [3]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025),https://arxiv.org/abs/2502.13923 25

    Pith/arXiv arXiv 2025

  4. [4]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Bärmann, L., Waibel, A.: Where did i leave my keys?-episodic-memory-based ques- tion answering on egocentric videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1560–1568 (2022) 11

    2022

  5. [5]

    Annotation Content Time Window: DAY1 13:11:19-13:11:20 Observation: The user is manipulating a rack part which suddenly falls, indicating instability and risk of impact injury

    Chen, J., Lv, Z., Wu, S., Lin, K.Q., Song, C., Gao, D., Liu, J.W., Gao, Z., Mao, D., Shou, M.Z.: Videollm-online: Online video large language model for streaming Vinci2 29 Time Window: DAY1 00:04:14-00:04:55 Observation: Stir-Stir the contents in the mug well Response: When you stir, keep the spoon mostly in the center and use small circles, then scrape a...

    2024

  6. [6]

    arXiv preprint arXiv:2412.05271 (2024) 12, 13

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024) 12, 13

    Pith/arXiv arXiv 2024

  7. [7]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cheng, S., Guo, Z., Wu, J., Fang, K., Li, P., Liu, H., Liu, Y.: Egothink: Evaluating first-person perspective thinking capability of vision-language models. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14291–14302 (2024) 3

    2024

  8. [8]

    arXiv preprint arXiv:2305.02750 (2023) 4

    Deng, Y., Lei, W., Lam, W., Chua, T.S.: A survey on proactive dialogue systems: Problems, methods, and prospects. arXiv preprint arXiv:2305.02750 (2023) 4

    Pith/arXiv arXiv 2023

  9. [9]

    Personal and ubiquitous computing 5(1), 4–7 (2001) 4

    Dey, A.K.: Understanding and using context. Personal and ubiquitous computing 5(1), 4–7 (2001) 4

    2001

  10. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Dong, Y., Kang, C., Zhang, J., Zhu, Z., Wang, Y., Yang, X., Su, H., Wei, X., Zhu, J.: Benchmarking robustness of 3d object detection to common corruptions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1022–1032 (2023) 3

    2023

  11. [11]

    In: European Conference on Computer Vision

    Fan, Y., Ma, X., Wu, R., Du, Y., Li, J., Gao, Z., Li, Q.: Videoagent: A memory- augmented multimodal agent for video understanding. In: European Conference on Computer Vision. pp. 75–92. Springer (2024) 4, 11

    2024

  12. [12]

    In: ICCV (2021) 3

    Girdhar, R., Grauman, K.: Anticipative video transformer. In: ICCV (2021) 3

    2021

  13. [13]

    In: CVPR (2022) 3

    Goyal, M., Modi, S., Goyal, R., Gupta, S.: Human hands as probes for interactive object understanding. In: CVPR (2022) 3

    2022

  14. [14]

    In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition

    Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Ham- burger, J., Jiang, H., Liu, M., Liu, X., et al.: Ego4d: Around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition. pp. 18995–19012 (2022) 3

    2022

  15. [15]

    arXiv preprint arXiv:2501.12948 (2025) 25, 27

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025) 25, 27

    Pith/arXiv arXiv 2025

  16. [16]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    He, B., Li, H., Jang, Y.K., Jia, M., Cao, X., Shah, A., Shrivastava, A., Lim, S.N.: Ma-lmm: Memory-augmented large multimodal model for long-term video under- standing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13504–13514 (2024) 4

    2024

  17. [17]

    arXiv preprint arXiv:2507.18342 (2025) 3

    He, Y., Huang, Y., Chen, G., Pei, B., Xu, J., Lu, T., Pang, J.: Egoexobench: A benchmark for first-and third-person view video understanding in mllms. arXiv preprint arXiv:2507.18342 (2025) 3

    Pith/arXiv arXiv 2025

  18. [18]

    IEEE Transactions on Image Processing29, 7795–7806 (2020) 3

    Huang, Y., Cai, M., Li, Z., Lu, F., Sato, Y.: Mutual context network for jointly estimating egocentric gaze and action. IEEE Transactions on Image Processing29, 7795–7806 (2020) 3

    2020

  19. [19]

    In: ECCV (2018) 3

    Huang, Y., Cai, M., Li, Z., Sato, Y.: Predicting gaze in egocentric video by learning task-dependent attention transition. In: ECCV (2018) 3

    2018

  20. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huang, Y., Chen, G., Xu, J., Zhang, M., Yang, L., Pei, B., Zhang, H., Dong, L., Wang, Y., Wang, L., et al.: Egoexolearn: A dataset for bridging asynchronous ego- and exo-centric view of procedural activities in real world. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22072– 22086 (2024) 3

    2024

  21. [21]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Huang, Y., Sugano, Y., Sato, Y.: Improving action segmentation via graph-based temporal reasoning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14024–14034 (2020) 3 Vinci2 33

    2020

  22. [22]

    arXiv preprint arXiv:2412.21080 (2024) 2, 4

    Huang, Y., Xu, J., Pei, B., He, Y., Chen, G., Yang, L., Chen, X., Wang, Y., Nie, Z., Liu, J., et al.: Vinci: A real-time embodied smart assistant based on egocentric vision-language model. arXiv preprint arXiv:2412.21080 (2024) 2, 4

    Pith/arXiv arXiv 2024

  23. [23]

    International Journal of Computer Vi- sion132(9), 3977–4002 (2024) 3

    Huang, Y., Yang, L., Chen, G., Zhang, H., Lu, F., Sato, Y.: Matching compound prototypes for few-shot action recognition. International Journal of Computer Vi- sion132(9), 3977–4002 (2024) 3

    2024

  24. [24]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Huang, Y., Yang, L., Sato, Y.: Weakly supervised temporal sentence grounding with uncertainty-guided self-training. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 18908–18918 (2023) 3

    2023

  25. [25]

    In: Findings of the Association for Computational Linguis- tics: ACL 2025

    Jeong, S., Kim, K., Baek, J., Hwang, S.J.: Videorag: Retrieval-augmented genera- tion over video corpus. In: Findings of the Association for Computational Linguis- tics: ACL 2025. pp. 21278–21298 (2025) 4

    2025

  26. [26]

    Advances in Neural Information Processing Systems35, 3343– 3360 (2022) 11

    Jia, B., Lei, T., Zhu, S.C., Huang, S.: Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems35, 3343– 3360 (2022) 11

    2022

  27. [27]

    arXiv preprint arXiv:2511.16221 (2025) 2

    Kang, C., Huang, Y., Ouyang, L., Zhang, M., Liu, R., Sato, Y.: Can mllms read the room? a multimodal benchmark for assessing deception in multi-party social interactions. arXiv preprint arXiv:2511.16221 (2025) 2

    arXiv 2025

  28. [28]

    In: Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology

    Lee, G., Xia, M., Numan, N., Qian, X., Li, D., Chen, Y., Kulshrestha, A., Chat- terjee, I., Zhang, Y., Manocha, D., et al.: Sensible agent: A framework for unob- trusive interaction with proactive ar agents. In: Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. pp. 1–22 (2025) 4

    2025

  29. [29]

    arXiv preprint arXiv:2408.03326 (2024) 12, 13, 14

    Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024) 12, 13, 14

    Pith/arXiv arXiv 2024

  30. [30]

    Science China Information Sciences 68(10), 200102 (2025) 2

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. Science China Information Sciences 68(10), 200102 (2025) 2

    2025

  31. [31]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al.: Mvbench: A comprehensive multi-modal video understanding benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22195–22206 (2024) 3, 14

    2024

  32. [32]

    In: Eu- ropean conference on computer vision

    Li, Y.M., Huang, W.J., Wang, A.L., Zeng, L.A., Meng, J.K., Zheng, W.S.: Egoexo- fitness: Towards egocentric and exocentric full-body action understanding. In: Eu- ropean conference on computer vision. pp. 363–382. Springer (2024) 3

    2024

  33. [33]

    In: Proceedings of the 2024 conference on empirical methods in natural language processing

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection. In: Proceedings of the 2024 conference on empirical methods in natural language processing. pp. 5971– 5984 (2024) 2

    2024

  34. [34]

    arXiv preprint arXiv:2411.03628 (2024) 15

    Lin, J., Fang, Z., Chen, C., Wan, Z., Luo, F., Li, P., Liu, Y., Sun, M.: Stream- ingbench: Assessing the gap for mllms to achieve streaming video understanding. arXiv preprint arXiv:2411.03628 (2024) 15

    Pith/arXiv arXiv 2024

  35. [35]

    arXiv preprint arXiv:2412.08646 (2024) 4

    Liu, J., Yu, Z., Lan, S., Wang, S., Fang, R., Kautz, J., Li, H., Alvare, J.M.: Stream- chat: Chatting with streaming video. arXiv preprint arXiv:2412.08646 (2024) 4

    Pith/arXiv arXiv 2024

  36. [36]

    arXiv preprint arXiv:2508.09736 (2025) 4

    Long, L., He, Y., Ye, W., Pan, Y., Lin, Y., Li, H., Zhao, J., Li, W.: Seeing, listening, remembering, and reasoning: A multimodal agent with long-term memory. arXiv preprint arXiv:2508.09736 (2025) 4

    arXiv 2025

  37. [37]

    arXiv preprint arXiv:2411.13093 (2024) 3, 4, 25 34 Sitong Gong et al

    Luo, Y., Zheng, X., Li, G., Yin, S., Lin, H., Fu, C., Huang, J., Ji, J., Chao, F., Luo, J., et al.: Video-rag: Visually-aligned retrieval-augmented long video compre- hension. arXiv preprint arXiv:2411.13093 (2024) 3, 4, 25 34 Sitong Gong et al

    arXiv 2024

  38. [38]

    Advances in Neural Information Processing Systems36, 46212–46244 (2023) 3, 11, 15

    Mangalam, K., Akshulakov, R., Malik, J.: Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems36, 46212–46244 (2023) 3, 11, 15

    2023

  39. [39]

    Niu, J., Li, Y., Miao, Z., Ge, C., Zhou, Y., He, Q., Dong, X., Duan, H., Ding, S., Qian, R., et al.: Ovo-bench: How far is your video-llms from real-world online video understanding? In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 18902–18913 (2025) 3, 4, 11, 15

    2025

  40. [40]

    OpenAI: Introducing GPT-4o and More Tools to ChatGPT Free Users.https: //openai.com(2024), accessed: 2026-03-06 13, 25, 27

    2024

  41. [41]

    OpenAI: GPT-5.1 (2025),https://openai.com, large language model 7, 11

    2025

  42. [42]

    Advances in Neural Information Pro- cessing Systems37, 135626–135679 (2024) 3, 5

    Peddi, R., Arya, S., Challa, B., Pallapothula, L., Vyas, A., Gouripeddi, B., Zhang, Q., Wang, J., Komaragiri, V., Ragan, E., et al.: Captaincook4d: A dataset for un- derstanding errors in procedural activities. Advances in Neural Information Pro- cessing Systems37, 135626–135679 (2024) 3, 5

    2024

  43. [43]

    arXiv preprint arXiv:2510.23569 (2025) 14

    Pei, B., Huang, Y., Xu, J., He, Y., Chen, G., Wu, F., Qiao, Y., Pang, J.: Ego- thinker: Unveiling egocentric reasoning with spatio-temporal cot. arXiv preprint arXiv:2510.23569 (2025) 14

    arXiv 2025

  44. [44]

    arXiv preprint arXiv:2308.07123 (2023) 3

    Plizzari, C., Goletto, G., Furnari, A., Bansal, S., Ragusa, F., Farinella, G.M., Damen, D., Tommasi, T.: An outlook into the future of egocentric vision. arXiv preprint arXiv:2308.07123 (2023) 3

    Pith/arXiv arXiv 2023

  45. [45]

    In: CVPR (2022) 3

    Plizzari, C., Planamente, M., Goletto, G., Cannici, M., Gusso, E., Matteucci, M., Caputo, B.: E2 (go) motion: Motion augmented event stream for egocentric action recognition. In: CVPR (2022) 3

    2022

  46. [46]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Qian, R., Ding, S., Dong, X., Zhang, P., Zang, Y., Cao, Y., Lin, D., Wang, J.: Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 24045–24055 (2025) 2, 4, 13

    2025

  47. [47]

    arXiv e-prints pp

    Qinghong Lin, K., Jinpeng Wang, A., Soldan, M., Wray, M., Yan, R., Zhong- congXu,E.,Gao,D.,Tu,R.,Zhao,W.,Kong,W.,etal.:Egocentricvideo-language pretraining. arXiv e-prints pp. arXiv–2206 (2022) 2, 3

    2022

  48. [48]

    In: ICCV (2023) 3

    Radevski, G., Grujicic, D., Blaschko, M., Moens, M.F., Tuytelaars, T.: Multimodal distillation for egocentric action recognition. In: ICCV (2023) 3

    2023

  49. [49]

    In: 1994 first workshop on mobile computing systems and applications

    Schilit, B., Adams, N., Want, R.: Context-aware computing applications. In: 1994 first workshop on mobile computing systems and applications. pp. 85–90. IEEE (1994) 4

    1994

  50. [50]

    In: CVPR (2020) 3

    Shan, D., Geng, J., Shu, M., Fouhey, D.: Understanding human hands in contact at internet scale. In: CVPR (2020) 3

    2020

  51. [51]

    arXiv preprint arXiv:2410.17434 (2024) 2, 13

    Shen, X., Xiong, Y., Zhao, C., Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., et al.: Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434 (2024) 2, 13

    Pith/arXiv arXiv 2024

  52. [52]

    arXiv preprint arXiv:2510.14032 (2025) 4

    Shen, X., Zhang, W., Chen, J., Elhoseiny, M.: Vgent: Graph-based retrieval- reasoning-augmented generation for long video understanding. arXiv preprint arXiv:2510.14032 (2025) 4

    arXiv 2025

  53. [53]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Shu, Y., Liu, Z., Zhang, P., Qin, M., Zhou, J., Liang, Z., Huang, T., Zhao, B.: Video-xl: Extra-long vision language model for hour-scale video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 26160–26169 (2025) 2

    2025

  54. [54]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al.: Moviechat: From dense token to sparse memory for long Vinci2 35 video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18221–18232 (2024) 2, 4

    2024

  55. [55]

    Starner, T.: Wearable computing and contextual awareness. Ph.D. thesis, Mas- sachusetts Institute of Technology (1999) 4

    1999

  56. [56]

    arXiv e-prints pp

    Tang, Y., Tang, H., Cao, T., Nguyen, L., Zhang, A., Cao, X., Liu, C., Ding, W., Li, Y.: Proagentbench: Evaluating llm agents for proactive assistance with real-world data. arXiv e-prints pp. arXiv–2602 (2026) 4

    2026

  57. [57]

    arXiv preprint arXiv:2312.11805 (2023) 15

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023) 15

    Pith/arXiv arXiv 2023

  58. [58]

    arXiv preprint arXiv:2505.05467 (2025) 2, 3, 4

    Wang, H., Feng, B., Lai, Z., Xu, M., Li, S., Ge, W., Dehghan, A., Cao, M., Huang, P.: Streambridge: Turning your offline video large language model into a proactive streaming assistant. arXiv preprint arXiv:2505.05467 (2025) 2, 3, 4

    arXiv 2025

  59. [59]

    In: ICCV (2023) 3

    Wang, H., Singh, M.K., Torresani, L.: Ego-only: Egocentric action detection with- out exocentric transferring. In: ICCV (2023) 3

    2023

  60. [60]

    In: European Conference on Computer Vision

    Wang, X., Zhang, Y., Zohar, O., Yeung-Levy, S.: Videoagent: Long-form video understanding with large language model as agent. In: European Conference on Computer Vision. pp. 58–76. Springer (2024) 25

    2024

  61. [61]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, X., Kwon, T., Rad, M., Pan, B., Chakraborty, I., Andrist, S., Bohus, D., Feniello, A., Tekin, B., Frujeri, F.V., et al.: Holoassist: an egocentric human in- teraction dataset for interactive ai assistants in the real world. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20270–20281 (2023) 3, 5

    2023

  62. [62]

    arXiv preprint arXiv:2411.179911(3), 5 (2024) 12

    Wang, Y., Meng, X., Wang, Y., Liang, J., Wei, J., Zhang, H., Zhao, D.: Videollm knows when to speak: Enhancing time-sensitive video comprehension with video- text duet interaction format. arXiv preprint arXiv:2411.179911(3), 5 (2024) 12

    arXiv 2024

  63. [63]

    arXiv preprint arXiv:2505.14668 (2025) 4

    Yang, B., Xu, L., Zeng, L., Liu, K., Jiang, S., Lu, W., Chen, H., Jiang, X., Xing, G., Yan, Z.: Contextagent: Context-aware proactive llm agents with open-world sensory perceptions. arXiv preprint arXiv:2505.14668 (2025) 4

    arXiv 2025

  64. [64]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Yang, J., Liu, S., Guo, H., Dong, Y., Zhang, X., Zhang, S., Wang, P., Zhou, Z., Xie, B., Wang, Z., et al.: Egolife: Towards egocentric life assistant. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 28885–28900 (2025) 2, 3, 5, 15

    2025

  65. [65]

    arXiv preprint arXiv:2408.01800 (2024) 12

    Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al.: Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 (2024) 12

    Pith/arXiv arXiv 2024

  66. [66]

    In: International Conference on Learning Representations (ICLR) (2025) 3

    Ye, H., Zhang, H., Daxberger, E., Chen, L., Lin, Z., Li, Y., Zhang, B., You, H., Xu, D., Gan, Z., et al.: MM-Ego: Towards building egocentric multimodal llms for video qa. In: International Conference on Learning Representations (ICLR) (2025) 3

    2025

  67. [67]

    arXiv preprint arXiv:2512.02425 (2025) 4

    Yeo, W., Kim, K., Yoon, J., Hwang, S.J.: Worldmm: Dynamic multimodal memory agent for long video reasoning. arXiv preprint arXiv:2512.02425 (2025) 4

    arXiv 2025

  68. [68]

    arXiv preprint arXiv:2501.13106 (2025) 14

    Zhang, B., Li, K., Cheng, Z., Hu, Z., Yuan, Y., Chen, G., Leng, S., Jiang, Y., Zhang, H., Li, X., et al.: Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106 (2025) 14

    Pith/arXiv arXiv 2025

  69. [69]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zhang, C., Yang, K., Hu, S., Wang, Z., Li, G., Sun, Y., Zhang, C., Zhang, Z., Liu, A., Zhu, S.C., et al.: Proagent: building proactive cooperative agents with large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 17591–17599 (2024) 2, 4 36 Sitong Gong et al

    2024

  70. [70]

    arXiv preprint arXiv:2406.08085 (2024) 2, 4

    Zhang, H., Wang, Y., Tang, Y., Liu, Y., Feng, J., Dai, J., Jin, X.: Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085 (2024) 2, 4

    Pith/arXiv arXiv 2024

  71. [71]

    arXiv preprint arXiv:2503.09143 (2025) 14

    Zhang, H., Chu, Q., Liu, M., Shi, H., Wang, Y., Nie, L.: Exo2ego: Exocen- tric knowledge guided mllm for egocentric video understanding. arXiv preprint arXiv:2503.09143 (2025) 14

    arXiv 2025

  72. [72]

    In: ECCV (2022) 3

    Zhang, L., Zhou, S., Stent, S., Shi, J.: Fine-grained egocentric hand-object seg- mentation: Dataset, model, and applications. In: ECCV (2022) 3

    2022

  73. [73]

    In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

    Zhang, Y., Dong, X.L., Lin, Z., Madotto, A., Kumar, A., Damavandi, B., Chai, J., Moon, S.: Proactive assistant dialogue generation from streaming egocentric videos. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 12055–12079 (2025) 2, 3, 4, 15

    2025

  74. [74]

    arXiv preprint arXiv:2510.14560 (2025) 2, 4, 10, 13

    Zhang, Y., Shi, C., Wang, Y., Yang, S.: Eyes wide open: Ego proactive video-llm for streaming video. arXiv preprint arXiv:2510.14560 (2025) 2, 4, 10, 13

    arXiv 2025

  75. [75]

    In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition

    Zhao, Y., Misra, I., Krähenbühl, P., Girdhar, R.: Learning video representations from large language models. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition. pp. 6586–6597 (2023) 3

    2023

  76. [76]

    Advances in neural information processing systems36, 46595–46623 (2023) 27

    Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems36, 46595–46623 (2023) 27

    2023

  77. [77]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Zohar, O., Wang, X., Dubois, Y., Mehta, N., Xiao, T., Hansen-Estruch, P., Yu, L., Wang, X., Juefei-Xu, F., Zhang, N., et al.: Apollo: An exploration of video understanding in large multimodal models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 18891–18901 (2025) 2

    2025