Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Current multimodal models can give step-by-step instructions for live tasks, but they consistently fail to spot user mistakes and issue corrective feedback in closed-loop streaming coaching.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:35 UTC pith:FUK6BPLV

load-bearing objection Useful multi-domain closed-loop streaming coaching benchmark with a real instruction-vs-correction gap; annotation/judge circularity is a real but not fatal soft spot. the 3 major comments →

arxiv 2607.02991 v1 pith:FUK6BPLV submitted 2026-07-03 cs.CV

GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video

Fang Liu , Jinpeng Chen , Ke Xu , Yuhao Liu , Huankang Guan , Xudong Lu , Bo Yang , Gerhard Hancke
show 2 more authors
Rui Liu Rynson W.H. Lau
This is my paper
classification cs.CV
keywords MLLMsstreaming videointeractive task guidanceprocedural coachingerror detectioncorrective feedbackclosed-loop interactionbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks how far multimodal large language models are from acting as real-time procedural coaches: systems that watch a user perform a multi-step task on a video stream, decide when to speak, detect mistakes, and steer the user back on track. To make that question measurable, the authors build GuideMe, a multi-domain streaming benchmark with 2,458 videos (223.7 hours) across cooking, object manipulation, daily-life guidance, and fitness, yielding 47,775 interaction samples that cover next-step instructions, completion feedback, error detection, and corrective guidance. They evaluate models with a three-part framework that jointly scores sequence-level temporal-semantic alignment, whether the model intervenes or stays silent at the right moments, and the quality of what it says. Across proprietary, open-source, and streaming-specialized models, the central finding is a sharp performance asymmetry: models are relatively capable at routine instruction, yet they systematically fail at identifying execution errors and providing corrective feedback. A sympathetic reader cares because success on offline video understanding does not transfer to the closed instruct-observe-correct loop that real assistance requires, and GuideMe turns that gap into a concrete training and evaluation target.

Core claim

Despite excelling at providing instructions, existing multimodal large language models consistently fail to identify execution errors and respond with corrective feedback when tested as real-time procedural coaches on streaming video. GuideMe establishes this asymmetry across diverse domains and model families: models can describe what should happen next, but they do not yet coach a user based on what is actually happening.

What carries the argument

GuideMe, a multi-domain streaming interaction benchmark, together with its three-component evaluation: temporal-semantic bipartite matching for sequence-level alignment of timed responses, behavioral classification of speak-versus-silent decisions at intervention anchors, and LLM-as-a-Judge scoring of content quality. The benchmark is produced by a three-stage pipeline that extracts correct, wrong, and correction actions, generates procedural knowledge, and turns those into timestamped dialogues.

Load-bearing premise

The automated annotation pipeline must produce trustworthy ground-truth intervention times and dialogues that truly represent closed-loop coaching, so that weak error-correction scores reflect model limits rather than label or metric artifacts.

What would settle it

Have independent human coaches re-annotate a large held-out subset of GuideMe error and correction events; if top models already match human timing and corrective content on those human labels, or if models trained only on clean next-step data succeed in live user sessions, the claimed instruct-versus-correct asymmetry would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reliable AI procedural coaches must jointly solve when to intervene, error detection, and corrective content—not only offline video description.
  • Model scale and streaming-oriented pretraining alone do not close the closed-loop gap on this benchmark.
  • Fine-tuning on GuideMe can improve temporal alignment and silence calibration while still leaving error correction weak.
  • Interactive-assistant evaluation must separately score silence-versus-speak decisions and corrective behavior, not only generic video QA.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training that only rewards clean next-step narration may under-prepare models for mistake-sensitive intervention.
  • The same instruct-observe-correct deficit is likely to appear in robot coaching and AR assistance that reuse the same multimodal backbones.
  • Synthetic error-and-recovery trajectories may be a necessary data primitive beyond expert demonstration videos.
  • Metrics that overweight correct silence under dense sampling can hide coaching failure; balanced intervention anchors matter for fair comparison.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces GuideMe, a multi-domain streaming benchmark for closed-loop procedural coaching with MLLMs. It aggregates 2,458 videos (223.7 hours) from EgoPER, CaptainCook4D, HoloAssist, and QEVD into 47,775 interaction samples spanning next-step instructions, completion feedback, error detection, and corrective guidance. A three-stage LLM-assisted pipeline extracts atomic actions, generates task knowledge via N=10 consensus, and produces timestamped dialogues. Evaluation uses temporal-semantic bipartite matching (sPrecision/sRecall/sF1), behavioral classification (CS/FA/NR/PC and Score), and LLM-as-a-Judge (Scorem). Across proprietary, open-source, and streaming models, results show reasonable instruction delivery but sharp degradation on error detection and correction (Tab. 2–3, Fig. 4), with ablations on sampling, history, window size, and interval (Tab. 4–5).

Significance. If the reported instruction–error asymmetry is real rather than an annotation artifact, GuideMe fills a clear gap relative to offline procedural datasets and general streaming benchmarks (Tab. 1). The multi-domain scale, explicit closed-loop framing, public code/data release, and multi-component protocol (bipartite matching + timing categories + content judge) are concrete contributions that can drive work on proactive intervention and recovery. The fine-tuning result and protocol ablations further make the resource usable for adaptation studies. The main scientific value is therefore as a diagnostic testbed that isolates when-to-speak and how-to-correct failures that offline video understanding does not stress.

major comments (3)
  1. Sec. 3.2 and Tab. 3: The central claim that models “excel at instructions but fail at error detection/correction” rests on LLM-generated Error/Correction references (task-graph deviation → correct next step for CaptainCook4D/EgoPER; HoloAssist corrections; then LLM dialogue generation) and on LLM-as-a-Judge scores for PC/Scorem (Sec. 3.5). No human inter-annotator agreement, preference study, or stratified quality audit is reported for the four response categories. If Error/Correction references are systematically more specific, less visually grounded, or stylistically harder than Instruction references, the Tab. 3 drops (e.g., Gemini 3 Pro sF1 42.0→33.8/34.3; Score 33.0→20.2/23.1) can partly be annotation/metric artifacts. A modest human validation subset (or category-wise difficulty controls) is load-bearing for the strongest claim.
  2. Sec. 3.4–3.5 and Tab. 2: Default evaluation supplies ground-truth dialogue history and uses Anchor-Based sampling (GT interventions + balanced silent timestamps). Tab. 4 shows that replacing GT history with model predictions sharply raises NR and lowers sF1/Score, and Dense sampling collapses behavior into silence or over-response. The paper correctly notes this, but the main tables still report the GT-history/Anchor setting as the primary evidence of “closed-loop” coaching failure. The manuscript should more clearly separate (i) single-step timing/content under oracle history from (ii) fully autonomous multi-turn closed-loop performance, and avoid overstating (i) as complete closed-loop coaching.
  3. Sec. 3.5, Eqs. (1)–(3) and behavioral Score: Soft-F1 and Score mix continuous embedding similarity, Gaussian temporal cost (σ free), silence handling, and LLM-judge quality into aggregate numbers. Sensitivity of σ, embedding model, and judge model is not reported; only window size and sampling interval are ablated (Tab. 5). Because the headline asymmetry is quantitative, a short sensitivity check (or fixed public judge prompt + seed) is needed so that ranking differences in Tab. 2–3 are not driven by unstated metric hyperparameters.
minor comments (5)
  1. Fig. 1 and abstract claim “closed-loop interaction where corrective guidance modifies the user’s subsequent actions,” but videos are pre-recorded; the loop is simulated via annotated recovery segments rather than live user adaptation. Clarify this distinction early.
  2. Table 1 lists “Closed-loop Interaction” only for GuideMe; a one-sentence definition of what counts as closed-loop (vs. timed feedback in QEVD/HoloAssist) would prevent over-reading the checkmark.
  3. Sec. 3.3: Train/test split sizes are given, but domain balance and error-rate balance across splits are not; a short table would help reproducibility of Tab. 3.
  4. Typographical/OCR artifacts appear in the provided text (e.g., “/enve♀e”, “/g♀behomepage”, “ofhow far”, “Wedesignaninferencepipeline”). Clean for camera-ready.
  5. Related work (Sec. 2.2–2.3) is thorough; still, briefly position against recent proactive streaming QA benchmarks on the specific instruct–observe–correct cycle rather than only general streaming understanding.

Circularity Check

1 steps flagged

No load-bearing circular derivation: GuideMe is an empirical benchmark whose main asymmetry claim is measured, not forced by definition; only mild LLM dual-use coupling in labels and judging.

specific steps
  1. other [Sec. 3.2 Conversation Generation; Sec. 3.5 LLM-as-a-Judge]
    "In the final stage, we generate instructional dialogues aligned with the atomic action boundaries, guided by the task description and generated procedural knowledge. ... After bipartite matching identifies the best-aligned prediction-reference pairs, we further evaluate the response quality of each matched pair using an LLM-as-a-Judge."

    Mild methodological self-reference only: LLM-generated dialogues supply reference text (including Error/Correction utterances derived from task-graph deviations), and an LLM judge later scores matched content (Scorem / PC quality). This couples model-family style into both labels and one quality metric, but does not force Instruction > Error by construction—categories and timestamps still come from external source labels, and sequence/timing metrics remain independent. Not a derivation that equals its inputs.

full rationale

GuideMe does not present a first-principles derivation or a fitted parameter renamed as a prediction. Its central claim—that MLLMs are stronger on next-step instructions than on error detection and corrective guidance—is an empirical observation on a constructed streaming benchmark (Tabs. 2–3, Fig. 4), not a quantity that reduces to its inputs by construction. Error/wrong/correction action categories and intervention timestamps are inherited from external source annotations and task graphs (EgoPER, CaptainCook4D, HoloAssist, QEVD; Sec. 3.2), which provide independent grounding for when an intervention should be instruction vs. error/correction. Temporal-semantic bipartite matching and behavioral CS/FA/NR/PC classification are embedding- and timing-based metrics that do not encode the instruction–error asymmetry by definition. Self-citations (e.g., PhoStream, Aura, X-Stream) appear as related streaming work and do not supply a uniqueness theorem or ansatz that forces the result. The only mild self-reference is that an LLM generates natural dialogues from structured actions and an LLM-as-a-Judge scores matched content quality (Sec. 3.2, 3.5); that is a validity/bias risk for Scorem and dialogue style, not a circular reduction of the claimed capability gap. Per the analyzer rules, that does not raise the score into the 4–6 range reserved for predictions forced by fit or self-citation chains. Honest finding: essentially non-circular empirical work (score 1 for the dual LLM role only).

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

As a benchmark paper, load-bearing content is definitional and methodological rather than physical axioms. The central claim rests on treating automated LLM-transformed dialogues as gold coaching behavior, on fixed inference hyperparameters (window, sampling), and on composite metrics that mix embedding similarity, silence decisions, and LLM judges. Free parameters are design choices that change measured scores (shown in ablations). Invented entities are the benchmark and evaluation stack themselves; independent evidence is the public data release and multi-model experiments, not external physical predictions.

free parameters (5)
  • sliding_window_seconds
    Default 60s visual context; Tab. 5 shows Score moves with 30s/120s choices and affects measured coaching ability.
  • anchor_based_vs_dense_sampling
    Default Anchor-Based balanced speak/silent queries; Dense sampling drastically changes CS/FA/Score (Tab. 4), so headline numbers depend on this protocol choice.
  • knowledge_generation_parallel_samples_N
    N=10 LLM samples with consensus refinement for task knowledge (Sec. 3.2); not fitted to model scores but shapes the reference procedures.
  • bipartite_matching_temporal_sigma
    Gaussian temporal cost Ldist uses a decay parameter σ (Eq. 2) that weights time alignment vs semantics in sF1.
  • LLM_judge_score_scale
    Integer 0–5 relevance/plausibility scores rescaled to 0–100 for Scorem and PC quality; judge model and rubric are free evaluation design choices.
axioms (5)
  • domain assumption Source dataset action graphs and human error labels correctly categorize correct, wrong, and correction atomic actions for coaching evaluation.
    Sec. 3.2 inherits labels from EgoPER, CaptainCook4D, HoloAssist, QEVD; the Error/Correction hardness claim depends on these categories being meaningful.
  • domain assumption An assistant under strict causal streaming with only past frames and dialogue history is the right formalization of real-time procedural coaching.
    Sec. 3.1 task definition; excludes offline full-video access and external triggers used in many QA benchmarks.
  • ad hoc to paper Temporal-semantic bipartite matching plus behavioral CS/FA/NR/PC plus LLM-as-a-Judge jointly measure closed-loop coaching quality.
    Sec. 3.5 introduces this three-component framework as the evaluation standard for GuideMe; conclusions about asymmetry are relative to these metrics.
  • domain assumption Sentence-transformer cosine similarity is an adequate soft textual alignment cost for guidance utterances.
    Eq. 1 defines Ltext via embeddings; mismatches in style or multi-intent corrections may be under/over-penalized.
  • domain assumption Standard MLLM zero-shot and streaming inference APIs can be compared fairly when given the same sliding window and (by default) GT dialogue history.
    Sec. 4.1 experimental setup; authors note GT history isolates single-step decisions from error accumulation.
invented entities (2)
  • GuideMe benchmark (multi-domain streaming closed-loop interaction samples) no independent evidence
    purpose: Provide train/test data and a unified protocol for next-step, completion, error detection, and corrective guidance under streaming constraints.
    Constructed artifact from prior videos via automated pipeline; not a physical entity. independent_evidence false beyond the paper’s release and experiments.
  • Three-component assessment framework (bipartite matching, behavioral classification, LLM-as-a-Judge) no independent evidence
    purpose: Score when models speak, what they say, and silence decisions without requiring token log-probs from closed APIs.
    Methodological construct defined in Sec. 3.5; results are only as meaningful as this construct.

pith-pipeline@v1.1.0-grok45 · 22013 in / 3793 out tokens · 34916 ms · 2026-07-12T05:35:28.146196+00:00 · methodology

0 comments
read the original abstract

While multimodal Large Language Models (MLLMs) excel at offline video understanding, an interesting question of how far they are from serving as a real-time procedural coach remains unknown. Such a role typically requires an MLLM to continuously monitor the execution, detect mistakes, and provide corrective guidance in a closed-loop interaction. In this paper, we construct GuideMe, the first multi-domain benchmark for streaming video that supports training and evaluation of MLLMs for closed-loop interactive task guidance. It comprises 2,458 videos spanning 223.7 hours across diverse domains (\eg, cooking, object manipulation, daily-life guidance, and fitness), with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance. To evaluate existing models on GuideMe, we design a three-component assessment framework to measure the capabilities of representative MLLMs, which consists of temporal-semantic bipartite matching for sequence-level alignment, behavioral classification for intervention timing, and LLM-as-a-Judge for content quality. Extensive experiments highlight a critical performance asymmetry: despite excelling at providing instructions, existing MLLMs consistently fail to identify execution errors and respond with corrective feedback. Code and data are released at https://fawnliu.github.io/project/guideme.

Figures

Figures reproduced from arXiv: 2607.02991 by Bo Yang, Fang Liu, Gerhard Hancke, Huankang Guan, Jinpeng Chen, Ke Xu, Rui Liu, Rynson W.H. Lau, Xudong Lu, Yuhao Liu.

Figure 1
Figure 1. Figure 1: Data examples in GuideMe. The benchmark captures the full closed￾loop interaction cycle: the assistant issues a step-level instruction, monitors the user’s execution, and upon detecting an error, provides corrective guidance to steer the user back on track. This cycle repeats throughout the procedure, enabling rigorous evaluation of interactive coaching ability. to the procedural task guidance: enabling an… view at source ↗
Figure 2
Figure 2. Figure 2: Annotation pipeline of GuideMe. The process integrates three stages: instructional activity extraction with error categorization, knowledge generation from the extracted activities, and conversation generation. 3.2 Annotation Pipeline We construct GuideMe through a three-stage automated annotation pipeline, illustrated in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Dataset statistics of GuideMe. (a) Scene Diversity: Distribution across seven categories, dominated by Home Life (38.4%) and Cooking (23.4%), with technical domains (Field Tech and IT Support) comprising 38.2%. (b) Duration Distribution: Temporal spread of 2,458 videos with a median of 3.6 minutes. The distribution includes long-form sequences up to 41.2 minutes, providing challenging samples for evaluatin… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative example. A microwave mug pizza task with two error-correction interventions. The ground-truth (GT) response is shown in orange alongside predictions from six representative models. routine instruction even when corrective feedback is needed. At 03:01, most responding models advance to later recipe steps instead of warning about the measuring spoon, while GPT-5.2 and MMDuet2 remain silent. At 05… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 12 linked inside Pith

  1. [2]

    arXiv preprint arXiv:2511.21631 (2025) 4, 10, 11

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, ...

    Pith/arXiv arXiv 2025

  2. [3]

    arXiv preprint arXiv:2502.13923 (2025) 10, 11, 14

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) 10, 11, 14

    Pith/arXiv arXiv 2025

  3. [4]

    Bao, Y., Yu, K., Zhang, Y., Storks, S., Bar-Yossef, I., de la Iglesia, A., Su, M., Zheng, X., Chai, J.: Can foundation models watch, talk and guide you step by step to make a cake? In: EMNLP. pp. 12325–12341 (2023) 5

    2023

  4. [5]

    In: WACV

    Ben-Shabat, Y., Yu, X., Saleh, F., Campbell, D., Rodriguez-Opazo, C., Li, H., Gould, S.: The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose. In: WACV. pp. 847–859 (2021) 4, 5

    2021

  5. [6]

    Available at Volcengine ARK Platform (2025) 3

    ByteDance: Doubao-seed-1.6. Available at Volcengine ARK Platform (2025) 3

    2025

  6. [7]

    Available at Volcengine ARK Platform (2026) 3, 10, 11, 12, 14

    ByteDance: Doubao-seed-1.8. Available at Volcengine ARK Platform (2026) 3, 10, 11, 12, 14

    2026

  7. [8]

    In: ECCV

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: ECCV. pp. 213–229. Springer (2020) 9

    2020

  8. [9]

    In: CVPR

    Chen, J., Lv, Z., Wu, S., Lin, K.Q., Song, C., Gao, D., Liu, J.W., Gao, Z., Mao, D., Shou, M.Z.: Videollm-online: Online video large language model for streaming video. In: CVPR. pp. 18407–18418 (2024) 4, 8, 10, 11

    2024

  9. [10]

    In: CVPR

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: CVPR. pp. 1290–1299 (2022) 9

    2022

  10. [11]

    IEEE TPAMI43(11), 4125–4141 (2020) 2, 4, 5

    Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al.: The epic-kitchens dataset: Collection, challenges and baselines. IEEE TPAMI43(11), 4125–4141 (2020) 2, 4, 5

    2020

  11. [12]

    In: ICCV

    Ding, X., Wu, H., Yang, Y., Jiang, S., Zhang, Q., Bai, D., Chen, Z., Cao, T.: Streammind: Unlocking full frame rate streaming video dialogue through event- gated cognition. In: ICCV. pp. 13448–13459 (2025) 4

    2025

  12. [13]

    In: CVPR

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: CVPR. pp. 24108–24118 (2025) 4 GuideMe 17

    2025

  13. [14]

    Fu, L., Yang, B., Kuang, Z., Song, J., Li, Y., Zhu, L., Luo, Q., Wang, X., Lu, H., Huang, M., Li, Z., Tang, G., Shan, B., Lin, C., Liu, Q., Wu, B., Feng, H., Liu, H., Huang, C., Tang, J., Chen, W., Jin, L., Liu, Y., Bai, X.: Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning (2024) 4

    2024

  14. [15]

    Available at Google DeepMind Model Cards (2025) 4, 10, 11, 12, 14

    Google DeepMind: Gemini 3 pro model card. Available at Google DeepMind Model Cards (2025) 4, 10, 11, 12, 14

    2025

  15. [16]

    Available at Google DeepMind Model Cards (2026) 1, 3, 10, 11

    Google DeepMind: Gemini 3.1 pro model card. Available at Google DeepMind Model Cards (2026) 1, 3, 10, 11

    2026

  16. [17]

    In: CVPR

    Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al.: Ego4d: Around the world in 3,000 hours of egocentric video. In: CVPR. pp. 18995–19012 (2022) 5

    2022

  17. [18]

    In: CVPR

    Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al.: Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In: CVPR. pp. 19383–19400 (2024) 5

    2024

  18. [19]

    In: CVPR

    Huang, Y., Chen, G., Xu, J., Zhang, M., Yang, L., Pei, B., Zhang, H., Dong, L., Wang, Y., Wang, L., et al.: Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. In: CVPR. pp. 22072–22086 (2024) 5

    2024

  19. [20]

    arXiv preprint arXiv:2410.21276 (2024) 1, 4

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024) 1, 4

    Pith/arXiv arXiv 2024

  20. [21]

    In: ICCVW

    Jang, Y., Sullivan, B., Ludwig, C., Gilchrist, I., Damen, D., Mayol-Cuevas, W.: Epic-tent: An egocentric video dataset for camping tent assembly. In: ICCVW. pp. 0–0 (2019) 4, 5

    2019

  21. [22]

    In: CVPR

    Lee, S.P., Lu, Z., Zhang, Z., Hoai, M., Elhamifar, E.: Error detection in egocentric procedural task videos. In: CVPR. pp. 18655–18666 (2024) 2, 4, 5, 6, 7

    2024

  22. [23]

    Lin,J.,Fang,Z.,Chen,C.,Wan,Z.,Luo,F.,Li,P.,Liu,Y.,Sun,M.:Streamingbench: Assessingthegapformllmstoachievestreamingvideounderstanding.arXivpreprint arXiv:2411.03628 (2024) 4, 8

    Pith/arXiv arXiv 2024

  23. [24]

    In: CVPR

    Lin, Y., Zhang, J., Wang, Q., Ye, H., Fu, Y., Liu, Y., Li, H.H., Chen, Y.: Hippomm: Hippocampal-inspired multimodal memory for long audiovisual event understanding. In: CVPR. pp. 5968–5977 (2026) 4

    2026

  24. [25]

    In: ICCV

    Liu, F., Liu, Y., Kong, Y., Xu, K., Zhang, L., Yin, B., Hancke, G., Lau, R.: Referring image segmentation using text supervision. In: ICCV. pp. 22124–22134 (2023) 4

    2023

  25. [26]

    In: CVPR

    Liu, F., Liu, Y., Xu, K., Hancke, G.P., Lau, R.W.: Gensplat: Bridging the gener- alization gap in 3dgs language comprehension. In: CVPR. pp. 5221–5231 (2026) 4

    2026

  26. [27]

    In: CVPR

    Liu, F., Liu, Y., Xu, K., Ye, S., Hancke, G.P., Lau, R.W.: Language-guided salient object ranking. In: CVPR. pp. 29803–29813 (2025) 4

    2025

  27. [28]

    In: ICLR (2026) 4

    Long, L., He, Y., Ye, W., Pan, Y., Lin, Y., Li, H., Zhao, J., Li, W.: Seeing, listening, remembering, and reasoning: A multimodal agent with long-term memory. In: ICLR (2026) 4

    2026

  28. [29]

    arXiv preprint arXiv:2604.04184 (2026) 8

    Lu, X., Bo, Y., Chen, J., Li, S., Guo, X., Guan, H., Liu, F., Xu, D., Sun, P., Sun, H., et al.: Aura: Always-on understanding and real-time assistance via video streams. arXiv preprint arXiv:2604.04184 (2026) 8

    Pith/arXiv arXiv 2026

  29. [30]

    In: ICML (2026) 4, 8 18 F

    Lu, X., Guan, H., Bo, Y., Chen, J., Guo, X., Li, S., Liu, F., Sun, P., Li, X., Zhang, W., et al.: Phostream: Benchmarking real-world streaming for omnimodal assistants in mobile scenarios. In: ICML (2026) 4, 8 18 F. Liu et al

    2026

  30. [31]

    In: WACV

    Mathew, M., Karatzas, D., Jawahar, C.: Docvqa: A dataset for vqa on document images. In: WACV. pp. 2200–2209 (2021) 4

    2021

  31. [32]

    In: ICCV

    Miech, A., Zhukov, D., Alayrac, J.B., Tapaswi, M., Laptev, I., Sivic, J.: Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In: ICCV. pp. 2630–2640 (2019) 2, 5

    2019

  32. [33]

    Niu, J., Li, Y., Miao, Z., Ge, C., Zhou, Y., He, Q., Dong, X., Duan, H., Ding, S., Qian, R., et al.: Ovo-bench: How far is your video-llms from real-world online video understanding? In: CVPR. pp. 18902–18913 (2025) 4, 8

    2025

  33. [34]

    NeurIPS37, 75853–75882 (2024) 2, 4, 5, 6, 7

    Panchal, S., Bhattacharyya, A., Berger, G., Mercier, A., Böhm, C., Dietrichkeit, F., Pourreza, R., Li, X., Madan, P., Lee, M., et al.: What to say and when to say it: Live fitness coaching as a testbed for situated interaction. NeurIPS37, 75853–75882 (2024) 2, 4, 5, 6, 7

    2024

  34. [35]

    NeurIPS37, 135626–135679 (2024) 2, 4, 5, 6, 7

    Peddi, R., Arya, S., Challa, B., Pallapothula, L., Vyas, A., Gouripeddi, B., Zhang, Q., Wang, J., Komaragiri, V., Ragan, E., et al.: Captaincook4d: A dataset for understanding errors in procedural activities. NeurIPS37, 135626–135679 (2024) 2, 4, 5, 6, 7

    2024

  35. [36]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Qian, R., Ding, S., Dong, X., Zhang, P., Zang, Y., Cao, Y., Lin, D., Wang, J.: Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 24045–24055 (2025) 4, 10, 11

    2025

  36. [37]

    In: CVPR

    Sener, F., Chatterjee, D., Shelepov, D., He, K., Singhania, D., Wang, R., Yao, A.: Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In: CVPR. pp. 21096–21106 (2022) 2, 4, 5

    2022

  37. [38]

    arXiv preprint arXiv:2601.03267 (2025) 10, 11, 14

    Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Ananthram, A., et al.: Openai gpt-5 system card. arXiv preprint arXiv:2601.03267 (2025) 10, 11, 14

    Pith/arXiv arXiv 2025

  38. [39]

    In: ICML (2026) 4

    Sun, P., Lang, S., Wu, D., Ding, Y., Feng, K., Liu, H., Ye, Z., Liu, R., Liu, Y.H., Wang, J., et al.: Spacevista: All-scale visual spatial reasoning from mm to km. In: ICML (2026) 4

    2026

  39. [40]

    In: ECCV (2026) 4

    Sun, P., Lu, X., Liu, H., Bo, Y., Wu, D., Guan, H., Cai, M., Chen, J., Guo, X., Li, S., et al.: X-stream: Exploring mllms as multiplexers for multi-stream understanding. In: ECCV (2026) 4

    2026

  40. [41]

    In: CVPR

    Tang, Y., Ding, D., Rao, Y., Zheng, Y., Zhang, D., Zhao, L., Lu, J., Zhou, J.: Coin: A large-scale dataset for comprehensive instructional video analysis. In: CVPR. pp. 1207–1216 (2019) 2, 4, 5

    2019

  41. [42]

    Team, Q.: Qwen3.5: Accelerating productivity with native multimodal agents (February 2026),https://qwen.ai/blog?id=qwen3.510, 11

    2026

  42. [43]

    arXiv preprint arXiv:2505.05467 (2025) 4

    Wang, H., Feng, B., Lai, Z., Xu, M., Li, S., Ge, W., Dehghan, A., Cao, M., Huang, P.: Streambridge: Turning your offline video large language model into a proactive streaming assistant. arXiv preprint arXiv:2505.05467 (2025) 4

    arXiv 2025

  43. [44]

    NeurIPS37, 95095– 95169 (2024) 4

    Wang, K., Pan, J., Shi, W., Lu, Z., Ren, H., Zhou, A., Zhan, M., Li, H.: Measuring multimodal mathematical reasoning with math-vision dataset. NeurIPS37, 95095– 95169 (2024) 4

    2024

  44. [45]

    5: Advancing open-source multimodal models in versatility, reasoning, and efficiency

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025) 4

    Pith/arXiv arXiv 2025

  45. [46]

    In: ICCV

    Wang, X., Kwon, T., Rad, M., Pan, B., Chakraborty, I., Andrist, S., Bohus, D., Feniello, A., Tekin, B., Frujeri, F.V., et al.: Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In: ICCV. pp. 20270–20281 (2023) 2, 4, 5, 6, 7 GuideMe 19

    2023

  46. [47]

    arXiv preprint arXiv:2512.06810 (2025) 3, 4, 11

    Wang, Y., Liu, S., Wang, D., Xu, N., Wan, G., Zhang, H., Zhao, D.: Mmduet2: Enhancing proactive interaction of video mllms with multi-turn reinforcement learning. arXiv preprint arXiv:2512.06810 (2025) 3, 4, 11

    arXiv 2025

  47. [48]

    arXiv preprint arXiv:2507.09313 (2025) 5, 8

    Wang, Y., Meng, X., Wang, Y., Zhang, H., Zhao, D.: Proactivevideoqa: A com- prehensive benchmark evaluating proactive interactions in video large language models. arXiv preprint arXiv:2507.09313 (2025) 5, 8

    Pith/arXiv arXiv 2025

  48. [49]

    In: CVPR

    Wang, Y., Wang, Y., Chen, B., Wu, T., Zhao, D., Zheng, Z.: Omnimmi: A compre- hensive multi-modal interaction benchmark in streaming video contexts. In: CVPR. pp. 18925–18935 (2025) 4

    2025

  49. [50]

    arXiv preprint arXiv:2412.10302 (2024) 4

    Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al.: Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 (2024) 4

    Pith/arXiv arXiv 2024

  50. [51]

    Xu, J., Guo, Z., Hu, H., Chu, Y., Wang, X., He, J., Wang, Y., Shi, X., He, T., Zhu, X., Lv, Y., Wang, Y., Guo, D., Wang, H., Ma, L., Zhang, P., Zhang, X., Hao, H., Guo, Z., Yang, B., Zhang, B., Ma, Z., Wei, X., Bai, S., Chen, K., Liu, X., Wang, P., Yang, M., Liu, D., Ren, X., Zheng, B., Men, R., Zhou, F., Yu, B., Yang, J., Yu, L., Zhou, J., Lin, J.: Qwen3...

    Pith/arXiv arXiv 2025

  51. [52]

    arXiv preprint arXiv:2510.09608 (2025) 4

    Xu, R., Xiao, G., Chen, Y., He, L., Peng, K., Lu, Y., Han, S.: Streamingvlm: Real-time understanding for infinite video streams. arXiv preprint arXiv:2510.09608 (2025) 4

    Pith/arXiv arXiv 2025

  52. [53]

    In: CVPR

    Yang, J., Liu, S., Guo, H., Dong, Y., Zhang, X., Zhang, S., Wang, P., Zhou, Z., Xie, B., Wang, Z., et al.: Egolife: Towards egocentric life assistant. In: CVPR. pp. 28885–28900 (2025) 5

    2025

  53. [54]

    arXiv preprint arXiv:2502.10810 (2025) 4

    Yang, Z., Hu, Y., Du, Z., Xue, D., Qian, S., Wu, J., Yang, F., Dong, W., Xu, C.: Svbench: A benchmark with temporal multi-turn dialogues for streaming video understanding. arXiv preprint arXiv:2502.10810 (2025) 4

    arXiv 2025

  54. [55]

    arXiv preprint arXiv:2511.05299 (2025) 4, 11

    Yang, Z., Zhang, K., Hu, Y., Wang, B., Qian, S., Wen, B., Yang, F., Gao, T., Dong, W., Xu, C.: Livestar: Live streaming assistant for real-world online video understanding. arXiv preprint arXiv:2511.05299 (2025) 4, 11

    arXiv 2025

  55. [56]

    arXiv preprint arXiv:2509.18154 (2025) 4

    Yu, T., Wang, Z., Wang, C., Huang, F., Ma, W., He, Z., Cai, T., Chen, W., Huang, Y., Zhao, Y., et al.: Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe. arXiv preprint arXiv:2509.18154 (2025) 4

    Pith/arXiv arXiv 2025

  56. [57]

    In: AAAI

    Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D.: Activitynet-qa: A dataset for understanding complex web videos via question answering. In: AAAI. pp. 9127–9134 (2019) 4

    2019

  57. [58]

    Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.W., Qiao, Y., et al.: Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In: ECCV. pp. 169–186. Springer (2024) 4

    2024

  58. [59]

    NeurIPS36, 46595–46623 (2023) 9, 10

    Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS36, 46595–46623 (2023) 9, 10

    2023

  59. [60]

    In: AAAI

    Zhou, L., Xu, C., Corso, J.: Towards automatic learning of procedures from web instructional videos. In: AAAI. vol. 32 (2018) 2, 4, 5

    2018