Pith.

Computer Vision and Pattern Recognition · arXiv:2611.17048 v1 · submitted 12 Nov 2026 · 50 references

AURA 2.0: From User-Driven Responses to Model-Initiated Conversations in Streaming Video

Shuhan Li, Huankang Guan, Xintong Guo, Yang Bo, Yuyang He, Dunyuan Xu, Jingyan Chen, Fang Liu

show 10 more authors

Tsz Kin Wan, Jinpeng Chen, Zhenqi He, Yuxuan Hu, Wenbo Li, Han Gao, Qijun Li, Long Chen, Hongsheng Li, Rui Liu

The paper says

“Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation.”

more of the abstract

To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) enables the model to invoke external tools using the current visual and conversational context and incorporate returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities. We open-source the model weights and inference code.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions.

6 major objections · 14 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 50 references say.

The paper says, at §7.3, Table 6

The review did not quote this passage.

Pith says: Objection 1 of 6 · major

The evaluation of Model-Initiated Conversation (MIC) is the central advance of the paper, but the evidence for temporal appropriateness is weak. The judge (gpt-4o) sees only two frames per message (one second before and at message onset). This is insufficient to verify that the initiation is event-grounded, non-repetitive, or non-interruptive.

more of objection 1

The Visual-Dialogue setting scores quality only inside a single human-annotated trigger window per video; messages outside this window are excluded from quality scoring but still count toward the msgs/min figure. With a median of 3.0-3.5 msgs/min over full playback, most unsolicited output bypasses the quality assessment. The paper acknowledges that the two settings are not directly comparable (Table 6 footnote) and that usefulness is subjective (§8), but the core claim that the model initiates conversations 'at appropriate moments' is not adequately supported by the reported protocol. The authors should either provide a more comprehensive temporal evaluation (e.g., human ratings of appropriateness on a per-message basis over full videos, or a control condition with event-free segments) or substantially qualify the claim.

The paper says, at §4.1, §7.3, §7.4

The review did not quote this passage.

Pith says: Objection 2 of 6 · major

There is a structural circularity concern: the training data for MIC, SATC, and IVG are generated by MLLMs (including decisions about when to initiate, what to say, tool calls, and bounding boxes), validated by MLLMs, and then evaluated using LLM judges (gpt-4o) that may share stylistic or content biases with the generator.

more of objection 2

The paper acknowledges MLLM-based verification (§4.1) but does not discuss the risk that the evaluation judges would reward behaviors aligned with the generator's preferences rather than genuine appropriateness or correctness. For MIC, the judge receives two frames and the conversational context; for SATC, an LLM judge compares answers against references. The authors should discuss this limitation explicitly and ideally provide a human evaluation or a held-out set of human-annotated examples to break the circularity.

The paper says, at §7.2, Tables 2-5

The review did not quote this passage.

Pith says: Objection 3 of 6 · major

The public benchmark results are reported without error bars, confidence intervals, or statistical significance tests. Many baseline numbers are author-run evaluations (e.g., JoyAI-VL-Interaction, MOSS-VL-Realtime on IPIBench, StreamArena, ProactiveVideoQA, PhoStream). The paper states that 'most baseline numbers are author-run evaluations with no error bars or significance tests' (reader's take).

more of objection 3

While this is common in technical reports, the lack of uncertainty quantification makes it difficult to assess whether the reported improvements are meaningful. The authors should at minimum report the number of runs and provide standard deviations or confidence intervals for their own model, and ideally for baselines where they control the evaluation.

The paper says, at §7.3, Table 6

The review did not quote this passage.

Pith says: Objection 4 of 6 · major

The trigger rate in the Visual-Only setting is 96.0% (24/25 videos), with a median of 3.5 msgs/min. Without a baseline (e.g., a random or always-speak policy, or a human speak rate), it is unclear whether this represents appropriate initiation or a learned prior to speak frequently.

more of objection 4

The paper notes that without dialogue context, the model 'may find it harder to distinguish observations worth sharing from those of limited value' (§7.3), but the evaluation does not include a control condition (e.g., videos with no noteworthy events) to test whether the model can remain silent when appropriate. The authors should provide such a control or explicitly acknowledge that the current protocol cannot distinguish over-triggering from appropriate initiation.

The paper says, at §5.2, Eq. (1)

The review did not quote this passage.

Pith says: Objection 5 of 6 · major

The response-control objective uses a weight of 0.25 for each of the two class-level mean focal losses and 1.0 for the other-token cross-entropy. The paper states that 'these coefficients are applied directly and are not renormalized by their sum' (§5.2).

more of objection 5

This means the total loss is L = 0.25 * (focal_silence + focal_respond) + 1.0 * (other_ce). The effective weight on the control tokens relative to other tokens is 0.25/1.0 = 0.25, but since the control losses are averaged over their respective sets (|S| and |R|) and the other loss is averaged over |O|, the balance also depends on the relative sizes of these sets. The paper does not analyze the sensitivity of the results to these hyperparameters or provide an ablation study. Given that the control tokens are the primary mechanism for learning when to speak or remain silent, the authors should justify the chosen coefficients or provide an ablation.

The paper says, at §6.1, §7.2.2, Table 3

The review did not quote this passage.

Pith says: Objection 6 of 6 · major

The Long-Horizon Interaction Memory (LHIM) is described as a key component for maintaining contextual continuity, but its contribution is not ablated in the main results. Table 3 shows that AURA 2.0 achieves 24.5% on Historical Retrospection (HR) compared to AURA's 22.7%, but it is unclear how much of this gain is due to LHIM versus other changes (e.g., improved base model, training data, MIC/SATC/IVG integration).

more of objection 6

The paper should include an ablation study that evaluates AURA 2.0 without LHIM on at least one benchmark (e.g., StreamArena HR or ProactiveVideoQA) to isolate its effect.

14 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 14

The abstract states that AURA 2.0 'achieves leading performance among open-source streaming models' on public benchmarks. This is supported by the results, but the phrase 'leading performance' could be more precise (e.g., 'state-of-the-art among open-source streaming models' or 'competitive with proprietary models on certain metrics').

The paper says, at §1, Figure 1

Pith says: minor objection 2 of 14

Figure 1 compares AURA 2.0 with MOSS-VL-Realtime and JoyAI-VL-Interaction across four benchmarks. The Tool use score for MOSS-VL-Realtime is plotted as zero because the model does not support tool calling. This is appropriate, but the figure caption should clarify that the zero is due to lack of capability, not poor performance.

show 12 more

The paper says, at §2.1, Table 1

Pith says: minor objection 3 of 14

Table 1 lists capabilities of representative streaming models. The cross (✗) for MIC, SATC, and IVG in prior models is described as 'supporting evidence was not identified, rather than a confirmed lack of capability.' This is a fair caveat, but the paper should also note that some proprietary models (e.g., Gemini Live) may support similar capabilities but are not included in the comparison due to lack of public documentation.

The paper says, at §4.1, Figure 3

Pith says: minor objection 4 of 14

Figure 3 describes the data construction pipeline. The text mentions that MLLM validators assess whether responses are 'supported by the available visual evidence and occur at appropriate times.' However, the criteria for 'appropriate times' are not specified. The authors should provide more detail on the validation prompts or rules used.

The paper says, at §4.2, Figure 4

Pith says: minor objection 5 of 14

Figure 4 shows the training data composition. The caption states that 'sample and token counts include replayed examples across stages and should not be interpreted as numbers of distinct source videos.' This is important and should also be stated in the main text where the dataset size is first mentioned (§1).

The paper says, at §5.2, Eq. (1)

Pith says: minor objection 6 of 14

The focal loss exponent γ is set to 2. The paper should cite the original focal loss paper (Lin et al., 2017) or provide a brief justification for this choice.

The paper says, at §6.1

Pith says: minor objection 7 of 14

The description of LHIM merging (level-1 to level-2 after 5 minutes, level-2 to level-3 after 10 minutes) is clear, but the paper does not specify the maximum number of levels or whether level-3 records are ever discarded. The text says 'Level-3 records remain in the context rather than being discarded to free capacity,' which suggests no upper bound. This could lead to unbounded context growth over very long sessions. The authors should discuss the practical limits.

The paper says, at §7.1, §7.2

Pith says: minor objection 8 of 14

The experimental setup for each benchmark specifies frame rates and visual context windows. However, the paper does not specify whether the same hyperparameters (e.g., temperature, top-p, max tokens) were used across all models for fair comparison. The authors should report decoding parameters.

The paper says, at §7.3, Figure 5

Pith says: minor objection 9 of 14

Figure 5 shows an example of MIC in the Visual-Dialogue Context setting. The example is illustrative, but the paper should clarify whether this is a cherry-picked example or representative of typical performance.

The paper says, at §7.4, Figure 6

Pith says: minor objection 10 of 14

Figure 6 shows SATC examples where tool use corrects errors. The paper should report the error rate of tool calls (e.g., how often the tool returns incorrect or irrelevant information) and how the model handles such cases.

The paper says, at §7.5, Table 7

Pith says: minor objection 11 of 14

The IVG evaluation uses 50 COCO images. This is a small diagnostic set. The paper should acknowledge that results may not generalize to video streams or more complex scenes. The comparison with Qwen3.5-9B is informative, but the backbone model is not designed for streaming visual grounding, so the improvement may partly reflect the training data rather than the streaming architecture.

The paper says, at §7.6, Figure 9

Pith says: minor objection 12 of 14

The inference latency optimization results are presented clearly. However, the paper does not report the memory usage or throughput under concurrent sessions, which would be relevant for practical deployment.

The paper says, at §8

Pith says: minor objection 13 of 14

The limitations section acknowledges that MIC usefulness is subjective and that the current system uses a cascaded ASR-VideoLLM-TTS pipeline. The paper could also mention the limitation that the evaluation of MIC relies on an internal benchmark with only 25 videos, which limits generalizability.

The paper says, at References

Pith says: minor objection 14 of 14

Several references are to arXiv preprints or technical reports (e.g., [2], [3], [5], [6], [7], [8], [9], [10], [11], [12], [22], [23], [24], [25], [26], [27], [37], [47], [49]). While this is acceptable for a fast-moving field, the paper should ensure that all claims attributed to these references are accurately represented and that the references are publicly accessible.

What would settle it

Using the released weights, re-score all MIC-Bench messages (the 173 Visual-Only messages and all dialogue-setting messages) with human raters who have full video access, labeling each as appropriately timed (novel, undiscussed visual event at onset, non-interruptive) or over-triggered/untimely, and compare with the two-frame gpt-4o quality scores; in the same protocol, run the checkpoint on about 20 event-free control videos (static scenes, no user queries) and record msgs/min. If human raters agree with the two-frame judge at 0.7 or higher (Cohen's kappa or ICC) and the control-set rate falls well below the reported 3.0-3.5 msgs/min, the 'appropriate moments' claim survives. If over-triggering exceeds about 20% of messages or the control rate stays near 1 msg/min, the MIC evidence must be treated as unverified until validated on an external or human-anchored benchmark.

from the skeptic's reading

How the review went

3 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 1 min 42 s · 25,461 tokens read, 14,690 written
  3. Referee major revision: 6 major objections, 14 minor 24 s · 25,904 tokens read, 2,967 written

Not run on this paper: the editor, the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The strongest objection

The central advance is MIC: the model must initiate conversations 'at appropriate moments without a user query' and remain silent when appropriate (abstract; Section 3.2). All direct evidence is internal MIC-Bench (25 videos, 50 test instances), judged by gpt-4o on two frames per message, sampled one second before and at message onset (Section 7.3). The defining property of the claim, temporal appropriateness of initiation, is not scorable from this protocol: a two-frame judge cannot verify that a message is event-grounded, non-repetitive, or non-interruptive. Further, the Visual-Dialogue setting scores quality only inside one human-annotated trigger window per video and excludes all other messages, while the model emits a median 3.0 msgs/min over full playback, so most unsolicited output bypasses quality scoring and affects only the uninterpreted msgs/min figure. The Visual-Only setting scores all 173 messages (quality 3.62/5, 15 Bad) but still without full temporal context. With no event-free control videos, no human speak-rate baseline, and no baseline model run under the MIC protocol, 3.0-3.5 msgs/min is compatible with a learned prior to speak frequently, a prior the Section 4.1 pipeline could plausibly instill (an MLLM decides when to open a conversation; an MLLM validator approves; an LLM judges). This sharpens the reader's weakest assumption: even a flawless judge would be blind here because appropriateness is never measured. I credit the human-annotated trigger windows and the annotator review of judge outputs, and the authors acknowledge MIC subjectivity (b385) and the 23.90% No Response rate (b280), but the temporal-appropriateness measurement gap is unacknowledged and unresolved.

I agree with the reader's conditional verdict and with locating the load-bearing soft spot in the MLLM synthesis/validation/judging loop; I place the sharpest failure one step further along. MIC's defining property is the timing of initiation ('at appropriate moments ... while remaining silent when appropriate', Section 3.2), and the MIC-Bench protocol cannot score this property: the gpt-4o judge receives only two frames per message, the dialogue-context setting scores only messages inside one human-annotated window per video, and messages outside those windows (most of the observed output at 3.0 msgs/min) are excluded from quality. The visual-only setting scores all 173 messages but still without full temporal context, yielding 3.62/5 with 15 Bad messages. Nothing in the paper anchors what an appropriate speak rate is, and no baseline model was run through MIC-Bench, so the observed rate is compatible with a disposition to speak frequently that could originate precisely from the Section 4.1 pipeline, where an MLLM decides when to initiate and an MLLM validator approves. I credit the paper for open-sourcing weights and code, for honestly acknowledging MIC subjectivity (b385), the 23.90% No Response rate (b280), and the non-comparability of the two MIC settings (b308), and for having human annotators define trigger windows and review judge outputs. Those acknowledgements cover preference subjectivity and missed responses, but not the temporal-appropriateness measurement gap itself. The proposed human full-context re-scoring plus an event-free control would settle whether the concern lands; if it lands, the flagship MIC claim is currently supported only at the level of 'trained to initiate often', which is weaker than 'initiates at appropriate moments'. Because the weights are public and the authors' conditions are checkable, CONDITIONAL remains the right verdict; I see no basis for REJECT or UNVERDICTED, and I do not elevate the public-benchmark parity questions (baseline configurations, rendering-fused table cells) above this concern, since those would at most require renumbering a claim the authors already phrase with care.

The referee's report

If the claims hold, AURA 2.0 represents a meaningful step beyond purely user-driven streaming assistants by enabling the model to proactively initiate conversations based on visual events and conversational context, without requiring a prior user query or standing instruction. This capability, combined with tool calling and visual grounding in a streaming setting, addresses a gap identified in the literature. The open-source release of model weights and inference code would facilitate reproducible research and practical applications. The scale of the training dataset and the three-stage curriculum are also notable engineering contributions. However, the significance is tempered by the reliance on internal benchmarks for the core MIC capability and the potential circularity in the training and evaluation pipeline (MLLM-generated data, MLLM validation, LLM judging).

Pith's reading of the paper (a machine's summary, not the authors')

Most AI assistants that watch video only answer when asked. AURA 2.0 is trained to also speak up on its own when something in the stream is worth mentioning, such as a truck ahead in traffic or a landmark related to an earlier conversation. The system adds three behaviors to an existing streaming video assistant: starting conversations without a prompt (MIC), calling external tools like web search or weather services while looking at the stream (SATC), and drawing boxes around objects it refers to (IVG). A long-horizon memory summarizes older video into text so the model can still refer to events that left its 30-second visual window.

The team built a large synthetic training set: about 11.4 billion tokens from roughly 5,400 hours of video. Another AI model generated clip descriptions, sample conversations, tool calls, and box coordinates, and a validator model filtered out bad examples. Training happened in three stages to avoid degrading the old question-answering behavior.

On four public streaming benchmarks the model reports the best results among open-source streaming models, and targeted tests show tools raising answer accuracy and grounding improving over the base model. The main caveat is that the new behaviors were tested on small internal sets: 25 videos for conversation initiation, 167 questions for tool use, and 50 images for grounding, with AI judges scoring quality. The weights and inference code are promised open source, but the training data and internal test sets are not released.

Core claim. The load-bearing assertion is that AURA 2.0 retains query-driven streaming interaction and adds Model-Initiated Conversation: the model can 'identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query' (abstract; Section 3.2). The paper further claims in Section 2.1 that no existing model is trained to initiate open-ended conversations without any prior request (sentence b49). If the paper is correct, an open-source streaming assistant can appropriately start unsolicited conversations while matching or beating prior open-source streaming models on four public streaming benchmarks.

The premise it rests on. The synthetic data pipeline of Section 4.1 assumes that MLLM-generated clip descriptions, conversation openings, tool traces, and bounding-box annotations, after MLLM-based verification, are correct supervision for the target behaviors. All three new capabilities are trained and then quality-checked by the same family of techniques: MLLM synthesis, MLLM validation, and, for MIC and SATC, LLM judging in Sections 7.3 and 7.4. If the generator MLLM systematically describes scenes, timings, or tool uses that do not match the actual video, the learned behaviors encode those biases, and the evaluation judges would share the same blind spot. This is structurally distinct from the claim itself: even a model that scores well on the released benchmarks could have learned behaviors misaligned with real streaming interactions.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) enables the model to invoke external tools using the current visual and conversational context and incorporate returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities. We open-source the model weights and inference code.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.99

“Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.09 · human participants 0.20 · controlled comparison 0.18 · randomized assignment 0.01 · states quantitative result 0.06 · compares with prior results 0.85 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. background 0.68 0.05 0.03 asserted 1.00
To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. contribution 1.00 0.88 0.05 asserted 0.98
The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. contribution 0.95 0.76 0.25 asserted 1.00
Stream-Aware Tool Calling (SATC) enables the model to invoke external tools using the current visual and conversational context and incorporate returned information into the ongoing interaction. contribution 0.69 0.81 0.04 asserted 1.00
Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. contribution 0.86 0.65 0.03 asserted 1.00
The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. method 0.61 0.75 0.03 asserted 0.95
To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. method 0.99 0.96 0.03 asserted 0.99
We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. method 0.59 0.92 0.04 asserted 0.99
Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities. result 1.00 0.95 0.64 asserted 1.00
We open-source the model weights and inference code. contribution 0.95 0.94 0.04 asserted 0.98

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 7,500 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (9)
Figure 1 of the paper
Figure 1. Comparison of AURA 2.0 with MOSS-VL-Realtime and JoyAI-VL-Interaction across four public streaming benchmarks. The Tool use score for MOSS-VL-Realtime on StreamArena is plotted as zero because the model does not support tool calling. Responses are triggered by user queries, with models responding either immediately or later when relevant evidence becomes available. However, events in the video stream may offer meaningful opportunities for conversation even without a user query. To move beyond purely user-driven interaction, we introduce Model-Initiated Conversation (MIC), which…
Figure 2 of the paper
Figure 2. Overview of the AURA 2.0 streaming interaction framework. The model incrementally processes video chunks together with user inputs and interaction history, producing either <|silent|> for silent observation or <|respond|> followed by a text response or tool call. Tool results are incorporated into the context to support subsequent responses. Dashed outlines indicate historical messages beyond the active visual window. Existing vision-language models support object and region localization in images and videos [38, 39, 40, 41]. In streaming interactions, spatial grounding requires…
Figure 3 of the paper
Figure 3. Data construction pipeline for AURA 2.0. The pipeline transforms collected videos into a verified streaming training mixture through three stages: (1) Video collection and preparation standardizes videos into clips and generates clip-level descriptions. (2) Task-specific data synthesis uses the clips and descriptions to construct General QA, MIC, SATC, and IVG trajectories. (3) Streaming assembly and quality assessment aligns the trajectories to video chunks and filters invalid or low-quality samples. activity. The prompt encourages useful contributions and discourages unsupported…
Figure 4 of the paper
Figure 4. Training data composition and response-control statistics across the three training stages. (a) Sample distribution by stage (inner ring) and task (outer ring). (b) Stage-wise sample and token allocations and silent/response decision proportions. (c) Task-wise sample counts and shares aggregated across stages. Together, these distributions characterize the task coverage and supervision balance of the training mixture. 4.2 Dataset Composition and Statistics The verified training mixture retains Real-Time QA, Proactive QA, and Multi-Response QA alongside MIC, SATC, IVG, and…
Figure 5 of the paper
Figure 5. MIC example in the Visual-Dialogue Context setting. Before playback, the user mentions wanting to get home early despite heavy traffic, and the assistant advises caution around big trucks. Without any further user query, AURA 2.0 initiates two messages: at t = 66 s it warns about a large Amazon truck ahead, and at t = 78 s it advises keeping a steady speed and a safe distance as traffic tightens. Both messages build on the user’s expressed concern and the assistant’s earlier advice. Each frame is shown one second before the corresponding message onset. judge outputs and found no…
Figure 6 of the paper
Figure 6. SATC examples from the internal benchmark. Top: WebSearch identifies the “Zaochake” reservation as a breakfast experience at the water market. Bottom: WebSearch uses character names from the subtitles to identify Brotherhood of Blades II. Both examples correct answers generated without tools
Figure 7 of the paper
Figure 7. Real-world weather query at t = 1 s. AURA 2.0 identifies Rome from the Colosseum and calls get_current_datetime and get_city_weather to retrieve the forecast. Without tools, the model reports that weather information is unavailable in the video; with tools, it provides the forecast for the visually identified city. We further present a real-world weather query in
Figure 8 of the paper
Figure 8. IVG examples with less salient and peripheral targets. AURA 2.0 (top, red boxes) localizes both the handbag and the partially visible bus. Qwen3.5-9B (bottom, blue boxes) incorrectly declares the handbag absent and produces no response for the bus. Both models accurately localize the remote control despite the salient cat. 7.6 Inference Efficiency All measurements are run on a server with two accelerators of about 140 GB memory each. The language model occupies one accelerator; speech recognition, speech synthesis, and codec decoding share the other. We stream videos from…
Figure 9 of the paper
Figure 9. Cumulative effect of the inference optimizations on median TTFP with a single session. Each row adds one optimization to the preceding configuration, and the full configuration reduces median TTFP by 54.6%. The right column reports the change relative to the previous row

Figures are the authors', reproduced from the paper.

References (50)
  1. [1] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. VideoLLM-Online: Online video large language model for streaming video. InProc. CVPR, pages 18407–18418, 2024 on Pith
  2. [2] Xudong Lu, Yang Bo, Jinpeng Chen, Shuhan Li, Xintong Guo, Huankang Guan, Fang Liu, Dunyuan Xu, Peiwen Sun, Heyang Sun, Rui Liu, and Hongsheng Li. AURA: Always-on understanding and real-time assistance via video streams.arXiv preprint arXiv:2604.04184, 2026 on PithPith's reviewarXiv
  3. [3] Dingyu Yao, Shuhuan Gu, Qingyi Si, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Naibin Gu, Zheng Lin, Weiping Wang, Nan Duan, and Jiaqi Wang. Harnessing streaming video in the wild. arXiv preprint arXiv:2606.08615, 2026 on PithPith's reviewarXiv
  4. [4] Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. InProc. CVPR, pages 24045–24055, 2025 on Pith
  5. [5] Yueqian Wang, Songxiang Liu, Disong Wang, Nuo Xu, Guanglu Wan, Huishuai Zhang, and Dongyan Zhao. MMDuet2: Enhancing proactive interaction of video mllms with multi-turn reinforcement learning.arXiv preprint arXiv:2512.06810, 2025 on PitharXiv
  6. [6] Yikai Zheng, Xin Ding, Yifan Yang, Shiqi Jiang, Hao Wu, Qianxi Zhang, Weijun Wang, Ting Cao, and Yunxin Liu. Em-Garde: A propose-match framework for proactive streaming video understanding.arXiv preprint arXiv:2603.19054, 2026 on PithPith's reviewarXiv
  7. [7] Zhenyu Yang, Kairui Zhang, Bing Wang, Shengsheng Qian, and Changsheng Xu. LiveStarPro: Proactive streaming video understanding with hierarchical memory for long-horizon streams. arXiv preprint arXiv:2606.17798, 2026 on PithPith's reviewarXiv
  8. [8] Xiangyu Zeng, Yuandong Yang, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Qingyi Si, Dingyu Yao, Changlian Ma, Haoran Chen, Xinyu Chen, Yansong Shi, Junhao Zhou, Yifei Li, Jun Zhang, Chuanyu Qin, Chenxu Yang, Xinlei Yu, Kun Ouyang, Yuchen Shao, Qianshan Wei, Changhai Zhou, Jun Gao, Jia... on PithPith's reviewarXiv
  9. [9] Xudong Lu, Huankang Guan, Yang Bo, Jinpeng Chen, Xintong Guo, Shuhan Li, Fang Liu, Peiwen Sun, Xueying Li, Wei Zhang, Xue Yang, Rui Liu, and Hongsheng Li. PhoStream: Benchmarking real-world streaming for omnimodal assistants in mobile scenarios.arXiv preprint arXiv:2601.22575, 2026 on PitharXiv
  10. [10] Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, and Dongyan Zhao. ProactiveV- ideoQA: A comprehensive benchmark evaluating proactive interactions in video large language models.arXiv preprint arXiv:2507.09313, 2025 on PithPith's reviewarXiv
  11. [11] Jinzhao Li, Yinuo Chen, Wenxuan Song, Yijia Lei, Yichi Zhang, Honglei Yan, Panwang Pan, and Miao Liu. IPIBench: Evaluating interactive proactive intelligence of MLLMs under continuous streams.arXiv preprint arXiv:2605.27074, 2026 on PithPith's reviewarXiv
  12. [12] Xichen Zhang, Guankai Li, Yinghao Zhu, Shijian Wang, Sitong Wu, Shaozuo Yu, Meng Chu, Yuan Lu, and Jiaya Jia. StreamArena: Toward continuous, interactive, and long-horizon agentic streaming video understanding.arXiv preprint arXiv:2608.05703, 2026 on PithPith's reviewarXiv
  13. [13] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024. 22 on PithPith's reviewarXiv
  14. [14] Qwen Team. Qwen3.8-Max: A new bar for coding and cowork. https://qwen.ai/blog?i d=qwen3.8, August 2026. Accessed 2026-10-23 on Pith
  15. [15] Google. Gemini 3.8 Flash. https://ai.google.dev/gemini-api/docs/models/gemin i-3.8-flash, 2026. Accessed 2026-10-23 on Pith
  16. [16] ByteDance Seed. Seed2.1. https://seed.bytedance.com/en/seed2_1 , 2026. Accessed 2026-10-23 on Pith
  17. [17] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-VStream: Memory-based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024 on PithPith's reviewarXiv
  18. [18] Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. LiveCC: Learning video llm with streaming speech transcription at scale. InProc. CVPR, pages 29083–29095, 2025 on Pith
  19. [19] Shenghao Fu, Qize Yang, Yuan-Ming Li, Yi-Xing Peng, Kun-Yu Lin, Xihan Wei, Jian-Fang Hu, Xiaohua Xie, and Wei-Shi Zheng. ViSpeak: Visual instruction feedback in streaming videos. In Proc. ICCV, pages 21778–21788, 2025 on Pith
  20. [20] Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, et al. TimeChat-Online: 80% visual tokens are naturally redundant in streaming videos. InProc. ACM MM, pages 10807–10816, 2025 on Pith
show 30 more
  1. [21] Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, and Ping Huang. StreamBridge: Turning your offline video large language model into a proactive streaming assistant.arXiv preprint arXiv:2505.05467, 2025 on PitharXiv
  2. [22] Weicai Yan, Yuhong Dai, Qi Ran, Haodong Li, Wang Lin, Tao Jin, Xing Xie, Hao Liao, and Jianxun Lian. Proact-VL: A proactive VideoLLM for real-time AI companions.arXiv preprint arXiv:2603.03447, 2026 on PithPith's reviewarXiv
  3. [23] Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Xiangyu Zeng, Yifei Li, Haowen Hou, Zheming Liang, Congcong Wang, Kaiwen Tuo, Jun Zhang, Yuhan Zhu, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, and Jiaqi Wang. JoyAI-VL- Interaction: Re... on PithPith's reviewarXiv
  4. [24] Pengyu Wang, Chenkun Tan, Shaojun Zhou, Qirui Zhou, Yanxin Chen, Xingyang He, Huazheng Zeng, Jijun Cheng, Chenghao Wang, Xiaomeng Qian, et al. MOSS-VL technical report.arXiv preprint arXiv:2608.15045, 2026 on PithPith's reviewarXiv
  5. [25] Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, et al. MiniCPM-o 4.5: Towards real-time full-duplex omni-modal interaction.arXiv preprint arXiv:2604.27393, 2026 on PithPith's reviewarXiv
  6. [26] ByteDance Seed. SeedRealtime audio-visual full-duplex LLM released: Toward omni-modal natural interaction. https://seed.bytedance.com/en/blog/seedrealtime-audio-v isual-full-duplex-llm-released-toward-omni-modal-natural-interaction , August 2026. Accessed 2026-10-23 on Pith
  7. [27] Qwen Team. Qwen3.8-Omni: Towards native omni-modal agents.arXiv preprint arXiv:2609.25611, 2026 on PithPith's reviewarXiv
  8. [28] Google. Gemini Live API overview and capabilities. https://ai.google.dev/gemini-a pi/docs/live-api, 2026. Accessed 2026-10-21. 23 on Pith
  9. [29] Google. Gemini Live API capabilities guide. https://ai.google.dev/gemini-api/doc s/live-api/capabilities, 2026. Accessed 2026-10-23 on Pith
  10. [30] Google. Gemini 3.8 Live. https://ai.google.dev/gemini-api/docs/models/gemin i-3.8-live, 2026. Accessed 2026-10-23 on Pith
  11. [31] Qwen Team. Qwen2.5-Omni technical report.arXiv preprint arXiv:2503.20215, 2025 on PithPith's reviewarXiv
  12. [32] Qwen Team. Qwen3-Omni technical report.arXiv preprint arXiv:2509.17765, 2025 on PithPith's reviewarXiv
  13. [33] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024 on PithPith's reviewarXiv
  14. [34] Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. StreamingBench: Assessing the gap for mllms to achieve streaming video understanding. arXiv preprint arXiv:2411.03628, 2024 on PithPith's reviewarXiv
  15. [35] Junbo Niu, Yifei Li, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. OVO-Bench: How far is your video-llms from real-world online video understanding? InProc. CVPR, pages 18902–18913, 2025 on Pith
  16. [36] Yuxuan Wang, Yueqian Wang, Bo Chen, Tong Wu, Dongyan Zhao, and Zilong Zheng. Omn- iMMI: A comprehensive multi-modal interaction benchmark in streaming video contexts. In Proc. CVPR, pages 18925–18935, 2025 on Pith
  17. [37] Thinking Machines Lab. Interaction models: A scalable approach to human-ai collaboration. Thinking Machines Lab: Connectionism, May 2026. https://thinkingmachines.ai/blog/interaction- models/ on Pith
  18. [38] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025 on PithPith's reviewarXiv
  19. [39] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-VL technical report.arXiv preprint arXiv:2511.21631, 2025 on PithPith's reviewarXiv
  20. [40] Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025 on PithPith's reviewarXiv
  21. [41] Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, et al. Kwai Keye-VL 1.5 technical report.arXiv preprint arXiv:2509.01563, 2025 on PithPith's reviewarXiv
  22. [42] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2694–2703, 2023 on Pith
  23. [43] Seonguk Seo, Joon-Young Lee, and Bohyung Han. URVOS: Unified referring video object segmentation network with a large-scale benchmark. InComputer Vision – ECCV 2020, pages 208–223. Springer, 2020 on Pith
  24. [44] Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. DanceTrack: Multi-object tracking in uniform appearance and diverse motion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20993–21002, 2022. 24 on Pith
  25. [45] Yutao Cui, Chenkai Zeng, Xiaoyu Zhao, Yichun Yang, Gangshan Wu, and Limin Wang. SportsMOT: A large multi-object tracking dataset in multiple sports scenes. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9921–9931, 2023 on Pith
  26. [46] Qwen Team. Qwen3.5-9B model card. https://huggingface.co/Qwen/Qwen3.5-9B, 2026. Accessed 2026-10-23 on Pith
  27. [47] vLLM-Omni Team. vLLM-Omni technical report: A unified serving runtime for omni-modality generation.arXiv preprint arXiv:2610.09307, 2026 on PithPith's reviewarXiv
  28. [48] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. InComputer Vision – ECCV 2014, pages 740–755. Springer, 2014 on Pith
  29. [49] Xudong Lu, Xueying Li, Annan Wang, Yang Bo, Jinpeng Chen, Zengliang Li, Nianzu Yang, Rui Liu, Xue Yang, Jingwen Hou, and Hongsheng Li. OmniInteract: Benchmarking real-world streaming interaction for real-time omnimodal assistants. InProceedings of the 2026 Conference on Empiri... on Pith
  30. [2026] Association for Computational Linguistics. 25 on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment