ProactiveBench evaluates streaming models at one-second resolution without response cues, finding that premature responses dominate over missed responses for most systems.
Omn- immi: A comprehensive multi-modal interaction benchmark in streaming video contexts
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
abstract
The rapid advancement of multi-modal language models (MLLMs) like GPT-4o has propelled the development of Omni language models, designed to process and proactively respond to continuous streams of multi-modal data. Despite their potential, evaluating their real-world interactive capabilities in streaming video contexts remains a formidable challenge. In this work, we introduce OmniMMI, a comprehensive multi-modal interaction benchmark tailored for OmniLLMs in streaming video contexts. OmniMMI encompasses over 1,121 videos and 2,290 questions, addressing two critical yet underexplored challenges in existing video benchmarks: streaming video understanding and proactive reasoning, across six distinct subtasks. Moreover, we propose a novel framework, Multi-modal Multiplexing Modeling (M4), designed to enable an inference-efficient streaming model that can see, listen while generating.
citation-role summary
background 2
citation-polarity summary
roles
background 2representative citing papers
Egostream introduces a diagnostic benchmark that expands 2,250 questions into 8,528 recall-conditioned evaluations to measure streaming episodic memory performance across detail, spatial, temporal, event, social, causal, and prospective dimensions in egocentric vision.
Introduces the first dedicated benchmark for live multi-modal LLM task guidance with mistake detection and a streaming baseline model.
MOSS-VL, an open 11.3B vision-language model family, uses gated cross-attention and a real-time SFT stage to achieve best-in-class streaming-video scores and faster time-to-first-token as visual context grows.
ProactiveVideoQA is a benchmark for proactive video question answering, and the proposed PAUC metric jointly scores response timing and content, claiming better alignment with human preferences.
TRACE shows identical streaming-video QA accuracy can hide large differences in workload, completion, and answer validity, and that proactive quality separates from false alarms, misses, and response delay.
Aero Realtime aligns continuous video, audio, and text output on one 80ms grid, letting a 4B model generate lexical tokens or silence in a single stream while reusing the KV cache.
citing papers explorer
-
ProactiveBench: Can Streaming Video Models Really Interact Like Humans?
ProactiveBench evaluates streaming models at one-second resolution without response cues, finding that premature responses dominate over missed responses for most systems.
-
EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision
Egostream introduces a diagnostic benchmark that expands 2,250 questions into 8,528 recall-conditioned evaluations to measure streaming episodic memory performance across detail, spatial, temporal, event, social, causal, and prospective dimensions in egocentric vision.
-
Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?
Introduces the first dedicated benchmark for live multi-modal LLM task guidance with mistake detection and a streaming baseline model.
-
MOSS-VL Technical Report
MOSS-VL, an open 11.3B vision-language model family, uses gated cross-attention and a real-time SFT stage to achieve best-in-class streaming-video scores and faster time-to-first-token as visual context grows.
-
ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models
ProactiveVideoQA is a benchmark for proactive video question answering, and the proposed PAUC metric jointly scores response timing and content, claiming better alignment with human preferences.
-
TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding
TRACE shows identical streaming-video QA accuracy can hide large differences in workload, completion, and answer validity, and that proactive quality separates from false alarms, misses, and response delay.
-
Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation
Aero Realtime aligns continuous video, audio, and text output on one 80ms grid, letting a 4B model generate lexical tokens or silence in a single stream while reusing the KV cache.