{"id":"d9045abc-4012-4a0a-b052-91d7e36c7b35","arxiv_id":"2611.17048","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AURA 2.0 shows that a streaming video model can be trained to start useful conversations on its own, call tools, and ground answers in video frames, while beating open-source streaming baselines on public benchmarks.","lead":"AURA 2.0 is a video-understanding model that can start its own conversations, call external tools, and point at objects in the stream without waiting for a user to ask. It is one of the first open-source streaming models trained to speak up proactively, and it reports top scores among open-source rivals on four public streaming video benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The 'appropriate moments' half of the MIC claim is not measurable from the reported protocol: the judge sees two frames per message, extra-window messages are excluded from quality scoring, and 3.0-3.5 msgs/min is anchored to no baseline, so over-triggering cannot be distinguished from…","rationale":"I agree with the reader's conditional verdict and with locating the load-bearing soft spot in the MLLM synthesis/validation/judging loop; I place the sharpest failure one step further along. MIC's defining property is the timing of initiation ('at appropriate moments ... while remaining silent when appropriate', Section 3.2), and the MIC-Bench protocol cannot score this property: the gpt-4o judge receives only two frames per message, the dialogue-context setting scores only messages inside one human-annotated window per video, and messages outside those windows (most of the observed output at 3.0 msgs/min) are excluded from quality. The visual-only setting scores all 173 messages but still without full temporal context, yielding 3.62/5 with 15 Bad messages. Nothing in the paper anchors what an appropriate speak rate is, and no baseline model was run through MIC-Bench, so the observed rate is compatible with a disposition to speak frequently that could originate precisely from the Section 4.1 pipeline, where an MLLM decides when to initiate and an MLLM validator approves. I credit the paper for open-sourcing weights and code, for honestly acknowledging MIC subjectivity (b385), the 23.90% No Response rate (b280), and the non-comparability of the two MIC settings (b308), and for having human annotators define trigger windows and review judge outputs. Those acknowledgements cover preference subjectivity and missed responses, but not the temporal-appropriateness measurement gap itself. The proposed human full-context re-scoring plus an event-free control would settle whether the concern lands; if it lands, the flagship MIC claim is currently supported only at the level of 'trained to initiate often', which is weaker than 'initiates at appropriate moments'. Because the weights are public and the authors' conditions are checkable, CONDITIONAL remains the right verdict; I see no basis for REJECT or UNVERDICTED, and I do not elevate the public-benchmark parity questions (baseline configurations, rendering-fused table cells) above this concern, since those would at most require renumbering a claim the authors already phrase with care.","tokens_in":25461,"tokens_out":14690,"duration_ms":101871,"concrete_test":"Using the released weights, re-score all MIC-Bench messages (the 173 Visual-Only messages and all dialogue-setting messages) with human raters who have full video access, labeling each as appropriately timed (novel, undiscussed visual event at onset, non-interruptive) or over-triggered/untimely, and compare with the two-frame gpt-4o quality scores; in the same protocol, run the checkpoint on about 20 event-free control videos (static scenes, no user queries) and record msgs/min. If human raters agree with the two-frame judge at 0.7 or higher (Cohen's kappa or ICC) and the control-set rate falls well below the reported 3.0-3.5 msgs/min, the 'appropriate moments' claim survives. If over-triggering exceeds about 20% of messages or the control rate stays near 1 msg/min, the MIC evidence must be treated as unverified until validated on an external or human-anchored benchmark.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central advance is MIC: the model must initiate conversations 'at appropriate moments without a user query' and remain silent when appropriate (abstract; Section 3.2). All direct evidence is internal MIC-Bench (25 videos, 50 test instances), judged by gpt-4o on two frames per message, sampled one second before and at message onset (Section 7.3). The defining property of the claim, temporal appropriateness of initiation, is not scorable from this protocol: a two-frame judge cannot verify that a message is event-grounded, non-repetitive, or non-interruptive. Further, the Visual-Dialogue setting scores quality only inside one human-annotated trigger window per video and excludes all other messages, while the model emits a median 3.0 msgs/min over full playback, so most unsolicited output bypasses quality scoring and affects only the uninterpreted msgs/min figure. The Visual-Only setting scores all 173 messages (quality 3.62/5, 15 Bad) but still without full temporal context. With no event-free control videos, no human speak-rate baseline, and no baseline model run under the MIC protocol, 3.0-3.5 msgs/min is compatible with a learned prior to speak frequently, a prior the Section 4.1 pipeline could plausibly instill (an MLLM decides when to open a conversation; an MLLM validator approves; an LLM judges). This sharpens the reader's weakest assumption: even a flawless judge would be blind here because appropriateness is never measured. I credit the human-annotated trigger windows and the annotator review of judge outputs, and the authors acknowledge MIC subjectivity (b385) and the 23.90% No Response rate (b280), but the temporal-appropriateness measurement gap is unacknowledged and unresolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents AURA 2.0, a streaming video large language model that extends the prior AURA system with three new capabilities: Model-Initiated Conversation (MIC), Stream-Aware Tool Calling (SATC), and Interactive Visual Grounding (IVG). The central claim is that AURA 2.0 can identify topics worth discussing in an evolving video stream and initiate conversations at appropriate moments without a user query, while also supporting tool use and visual grounding. The authors construct a dedicated streaming interaction dataset (11.4B tokens, 5,400 hours of video) and adopt a three-stage training scheme with a focal-loss-based response-control objective. They incorporate Long-Horizon Interaction Memory (LHIM) for extended context. Evaluations on four public streaming benchmarks (PhoStream, ProactiveVideoQA, IPIBench, StreamArena) show leading performance among open-source streaming models. Targeted internal evaluations assess MIC, SATC, and IVG. The model weights and inference code are open-sourced.","tokens_in":25904,"tokens_out":2967,"duration_ms":23604,"significance":"If the claims hold, AURA 2.0 represents a meaningful step beyond purely user-driven streaming assistants by enabling the model to proactively initiate conversations based on visual events and conversational context, without requiring a prior user query or standing instruction. This capability, combined with tool calling and visual grounding in a streaming setting, addresses a gap identified in the literature. The open-source release of model weights and inference code would facilitate reproducible research and practical applications. The scale of the training dataset and the three-stage curriculum are also notable engineering contributions. However, the significance is tempered by the reliance on internal benchmarks for the core MIC capability and the potential circularity in the training and evaluation pipeline (MLLM-generated data, MLLM validation, LLM judging).","major_comments":[{"comment":"The evaluation of Model-Initiated Conversation (MIC) is the central advance of the paper, but the evidence for temporal appropriateness is weak. The judge (gpt-4o) sees only two frames per message (one second before and at message onset). This is insufficient to verify that the initiation is event-grounded, non-repetitive, or non-interruptive. The Visual-Dialogue setting scores quality only inside a single human-annotated trigger window per video; messages outside this window are excluded from quality scoring but still count toward the msgs/min figure. With a median of 3.0-3.5 msgs/min over full playback, most unsolicited output bypasses the quality assessment. The paper acknowledges that the two settings are not directly comparable (Table 6 footnote) and that usefulness is subjective (§8), but the core claim that the model initiates conversations 'at appropriate moments' is not adequately supported by the reported protocol. The authors should either provide a more comprehensive temporal evaluation (e.g., human ratings of appropriateness on a per-message basis over full videos, or a control condition with event-free segments) or substantially qualify the claim.","section":"§7.3, Table 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"There is a structural circularity concern: the training data for MIC, SATC, and IVG are generated by MLLMs (including decisions about when to initiate, what to say, tool calls, and bounding boxes), validated by MLLMs, and then evaluated using LLM judges (gpt-4o) that may share stylistic or content biases with the generator. The paper acknowledges MLLM-based verification (§4.1) but does not discuss the risk that the evaluation judges would reward behaviors aligned with the generator's preferences rather than genuine appropriateness or correctness. For MIC, the judge receives two frames and the conversational context; for SATC, an LLM judge compares answers against references. The authors should discuss this limitation explicitly and ideally provide a human evaluation or a held-out set of human-annotated examples to break the circularity.","section":"§4.1, §7.3, §7.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The public benchmark results are reported without error bars, confidence intervals, or statistical significance tests. Many baseline numbers are author-run evaluations (e.g., JoyAI-VL-Interaction, MOSS-VL-Realtime on IPIBench, StreamArena, ProactiveVideoQA, PhoStream). The paper states that 'most baseline numbers are author-run evaluations with no error bars or significance tests' (reader's take). While this is common in technical reports, the lack of uncertainty quantification makes it difficult to assess whether the reported improvements are meaningful. The authors should at minimum report the number of runs and provide standard deviations or confidence intervals for their own model, and ideally for baselines where they control the evaluation.","section":"§7.2, Tables 2-5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The trigger rate in the Visual-Only setting is 96.0% (24/25 videos), with a median of 3.5 msgs/min. Without a baseline (e.g., a random or always-speak policy, or a human speak rate), it is unclear whether this represents appropriate initiation or a learned prior to speak frequently. The paper notes that without dialogue context, the model 'may find it harder to distinguish observations worth sharing from those of limited value' (§7.3), but the evaluation does not include a control condition (e.g., videos with no noteworthy events) to test whether the model can remain silent when appropriate. The authors should provide such a control or explicitly acknowledge that the current protocol cannot distinguish over-triggering from appropriate initiation.","section":"§7.3, Table 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The response-control objective uses a weight of 0.25 for each of the two class-level mean focal losses and 1.0 for the other-token cross-entropy. The paper states that 'these coefficients are applied directly and are not renormalized by their sum' (§5.2). This means the total loss is L = 0.25 * (focal_silence + focal_respond) + 1.0 * (other_ce). The effective weight on the control tokens relative to other tokens is 0.25/1.0 = 0.25, but since the control losses are averaged over their respective sets (|S| and |R|) and the other loss is averaged over |O|, the balance also depends on the relative sizes of these sets. The paper does not analyze the sensitivity of the results to these hyperparameters or provide an ablation study. Given that the control tokens are the primary mechanism for learning when to speak or remain silent, the authors should justify the chosen coefficients or provide an ablation.","section":"§5.2, Eq. (1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Long-Horizon Interaction Memory (LHIM) is described as a key component for maintaining contextual continuity, but its contribution is not ablated in the main results. Table 3 shows that AURA 2.0 achieves 24.5% on Historical Retrospection (HR) compared to AURA's 22.7%, but it is unclear how much of this gain is due to LHIM versus other changes (e.g., improved base model, training data, MIC/SATC/IVG integration). The paper should include an ablation study that evaluates AURA 2.0 without LHIM on at least one benchmark (e.g., StreamArena HR or ProactiveVideoQA) to isolate its effect.","section":"§6.1, §7.2.2, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract states that AURA 2.0 'achieves leading performance among open-source streaming models' on public benchmarks. This is supported by the results, but the phrase 'leading performance' could be more precise (e.g., 'state-of-the-art among open-source streaming models' or 'competitive with proprietary models on certain metrics').","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 1 compares AURA 2.0 with MOSS-VL-Realtime and JoyAI-VL-Interaction across four benchmarks. The Tool use score for MOSS-VL-Realtime is plotted as zero because the model does not support tool calling. This is appropriate, but the figure caption should clarify that the zero is due to lack of capability, not poor performance.","section":"§1, Figure 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 1 lists capabilities of representative streaming models. The cross (✗) for MIC, SATC, and IVG in prior models is described as 'supporting evidence was not identified, rather than a confirmed lack of capability.' This is a fair caveat, but the paper should also note that some proprietary models (e.g., Gemini Live) may support similar capabilities but are not included in the comparison due to lack of public documentation.","section":"§2.1, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 3 describes the data construction pipeline. The text mentions that MLLM validators assess whether responses are 'supported by the available visual evidence and occur at appropriate times.' However, the criteria for 'appropriate times' are not specified. The authors should provide more detail on the validation prompts or rules used.","section":"§4.1, Figure 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 4 shows the training data composition. The caption states that 'sample and token counts include replayed examples across stages and should not be interpreted as numbers of distinct source videos.' This is important and should also be stated in the main text where the dataset size is first mentioned (§1).","section":"§4.2, Figure 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The focal loss exponent γ is set to 2. The paper should cite the original focal loss paper (Lin et al., 2017) or provide a brief justification for this choice.","section":"§5.2, Eq. (1)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The description of LHIM merging (level-1 to level-2 after 5 minutes, level-2 to level-3 after 10 minutes) is clear, but the paper does not specify the maximum number of levels or whether level-3 records are ever discarded. The text says 'Level-3 records remain in the context rather than being discarded to free capacity,' which suggests no upper bound. This could lead to unbounded context growth over very long sessions. The authors should discuss the practical limits.","section":"§6.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The experimental setup for each benchmark specifies frame rates and visual context windows. However, the paper does not specify whether the same hyperparameters (e.g., temperature, top-p, max tokens) were used across all models for fair comparison. The authors should report decoding parameters.","section":"§7.1, §7.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 5 shows an example of MIC in the Visual-Dialogue Context setting. The example is illustrative, but the paper should clarify whether this is a cherry-picked example or representative of typical performance.","section":"§7.3, Figure 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 6 shows SATC examples where tool use corrects errors. The paper should report the error rate of tool calls (e.g., how often the tool returns incorrect or irrelevant information) and how the model handles such cases.","section":"§7.4, Figure 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The IVG evaluation uses 50 COCO images. This is a small diagnostic set. The paper should acknowledge that results may not generalize to video streams or more complex scenes. The comparison with Qwen3.5-9B is informative, but the backbone model is not designed for streaming visual grounding, so the improvement may partly reflect the training data rather than the streaming architecture.","section":"§7.5, Table 7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The inference latency optimization results are presented clearly. However, the paper does not report the memory usage or throughput under concurrent sessions, which would be relevant for practical deployment.","section":"§7.6, Figure 9","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The limitations section acknowledges that MIC usefulness is subjective and that the current system uses a cascaded ASR-VideoLLM-TTS pipeline. The paper could also mention the limitation that the evaluation of MIC relies on an internal benchmark with only 25 videos, which limits generalizability.","section":"§8","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Several references are to arXiv preprints or technical reports (e.g., [2], [3], [5], [6], [7], [8], [9], [10], [11], [12], [22], [23], [24], [25], [26], [27], [37], [47], [49]). While this is acceptable for a fast-moving field, the paper should ensure that all claims attributed to these references are accurately represented and that the references are publicly accessible.","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":null,"pith_extraction":null,"created_at":"2026-11-13T07:14:16.085258+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}