{"id":"3c1d6c66-0785-4185-bbdb-a7e7d42ee300","arxiv_id":"2608.05703","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"StreamArena evaluates streaming video agents on hour-scale, open-ended tasks; StreamMind, a decoupled frontend-backend architecture, outperforms current streaming baselines across all four capabilities.","lead":"StreamArena is a new benchmark that tests multimodal agents on hour-long videos with open-ended questions about real-time perception, memory, proactive alerts, and external search. The authors also introduce StreamMind, a two-tier architecture that combines fast frontend interaction with asynchronous memory and retrieval, and report large gains over existing streaming systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tool-utilization scores do not verify tool invocation; correct Tool answers can come from the LLM's parametric memory, so the 228.1% Tool improvement may not measure tool use.","rationale":"The reader identifies the replay-harness unfairness in Appendix C.1 and Table 10 as the weakest assumption. That is a genuine concern: always-on baselines such as AURA and MiniCPM-o are evaluated through query-triggered replay with no hidden video state, which may understate their continuous-ingestion capabilities and directly affects all four comparative numbers. I focus instead on the Tool metric because it is a validity threat to the largest capability-specific gain, 228.1%, and to the claim of ranking first on all four capabilities. Unlike harness fairness, the Tool issue is cleanly testable with the released code by a no-tool ablation. The benchmark has real strengths: 243 hour-scale videos, open-ended human-validated questions, causal query and evidence timestamps, and released data and code. StreamMind's latency reduction from persistent state, 66.2%, is a system contribution independent of the Tool metric. However, the central claim bundles 'multimodal tool utilization' with the other capabilities. If correct answers can be produced from parametric memory without tool use, the reported Tool numbers cannot distinguish systems that use tools well from systems that merely know facts. The paper is transparent about not verifying invocation, but transparency does not make the metric valid. This does not collapse the paper; it makes the verdict conditional. The reader's harness concern remains and compounds the comparative issue, but the Tool metric is the most load-bearing single gap. Given the benchmark value and the feasibility of fixing the metric, I do not change the reader's CONDITIONAL verdict, hence UNCHANGED.","tokens_in":19203,"tokens_out":5247,"duration_ms":58132,"concrete_test":"Take the 1,732 Tool questions and run StreamMind in an ablated no-tool configuration: same Qwen3.5-397B-A17B backbone, same 128 frames and ASR, but disable Search, Recall, and all tool endpoints so the model must answer from the stream and parametric memory alone. Compare this no-tool accuracy with the reported 56.1% Tool accuracy. Also instrument the full StreamMind run to count correct answers whose transcript contains at least one search result that actually contributed to the final answer. If the no-tool accuracy is within about 80% of 56.1%, or if many correct answers lack any contributing tool call, the Tool capability claim is not measuring tool utilization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 defines Tool accuracy via Eq. (1), a factual-core answer match, and explicitly states that 'answer correctness alone does not establish whether or how a tool was invoked.' Appendix C.4 reinforces this for ThinkStream: its Tool score 'measures answer correctness without guaranteed tool invocation.' Yet the central claim rests on StreamMind 'ranks first among streaming systems on all four capabilities,' including a 228.1% relative Tool improvement over the strongest streaming baseline. Tool questions ask for facts unavailable from the stream, such as actor parent nationalities or birthdays. Such facts are often in the backbone's parametric memory, so a correct answer can be produced with zero tool calls. Without logging or verifying Search/Router tool invocations, 'Tool' accuracy is not a measure of tool utilization; it partly measures world-knowledge recall. This affects one of the four headline capabilities and the largest reported gain, so the central claim is not fully supported. Fixing the metric, e.g., by requiring a traceable tool call that contributes to the answer, is feasible and would not undermine the benchmark artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces StreamArena, a benchmark for hour-scale interactive streaming video understanding, containing 243 full-length videos (88.8 minutes on average) and 3,646 open-ended, human-validated question-answer pairs with separate query and evidence timestamps. Four capabilities are targeted: real-time multimodal perception, historical retrospection, proactive interaction, and multimodal tool utilization. The paper also proposes StreamMind, a two-tier architecture in which latency-critical frontend workers handle interaction and monitoring while backend workers asynchronously build persistent multimodal memory and perform recall and external search. Experiments compare offline turn-based models, recent-window methods, text-summary methods, model-internal compression methods, and StreamMind, reporting that StreamMind ranks first among streaming systems on all four capabilities, with relative improvements of 53.7% to 228.1% over the strongest corresponding streaming baselines and a 66.2% pooled query-to-answer latency reduction under a shared Qwen3.5-397B-A17B backbone.","tokens_in":19508,"tokens_out":5819,"duration_ms":61720,"significance":"If the main comparative claim holds, StreamArena is a valuable addition to the streaming-video evaluation landscape: it uses hour-scale videos, open-ended answers rather than multiple choice, dual temporal grounding for queries and evidence, and a four-capability coverage that existing benchmarks do not jointly provide. The paper also ships substantial artifacts, including the annotation data under CC BY 4.0, evaluation harness code, baseline adapters, and the StreamMind implementation, and it explicitly documents configuration details and reproducibility settings. The diagnostic analysis of modality availability, frame budgets, resolution, and reasoning mode is useful and appropriately caveated. My assessment is conditional, however, because the central comparative claim rests on an unequal evaluation harness for the streaming baselines, the Tool metric does not verify tool invocation despite being presented as a tool-utilization capability, and all quantitative results come from a single run with no uncertainty estimates.","major_comments":[{"comment":"The headline claim that StreamMind outperforms 'existing streaming baselines' is evaluated under an unequal harness. StreamMind ingests the stream continuously with persistent state (Eq. (11)-(12)), while every baseline is evaluated through query-triggered replay that reconstructs a causal prefix or recent window per task and carries no hidden video state across questions, as stated in Appendix C.1 and Table 10. For always-on systems such as AURA and MiniCPM-o-4.5, this may understate their capabilities: for example, MiniCPM-o's reactive protocol in Appendix C.3 is a half-duplex recent-window pass, not the native full-duplex sliding-context interface used for proactive tasks. Because the abstract and contributions claim improvement over streaming baselines, the comparison should either run these baselines in a continuous, stateful mode where such a mode exists, or the claims should be explicitly restricted to query-triggered replay baselines. This is load-bearing for all four headline improvement percentages.","section":"Section 3.1, Appendix C.1, Table 10"},{"comment":"The Tool metric is not a measure of tool utilization. Section 2.1 explicitly states that 'answer correctness alone does not establish whether or how a tool was invoked,' and Appendix C.4 notes for ThinkStream that its Tool score 'measures answer correctness without guaranteed tool invocation.' Yet Table 3 and Section 3.2 present Tool as one of the four capabilities and headline a 228.1% relative improvement. Because Tool questions ask for facts such as actor parent nationalities that may already be in the backbone's parametric memory, a correct answer can be produced with zero tool calls. One of the four headline capabilities and the largest reported gain are therefore not actually validated as tool use. The metric should require a traceable tool invocation whose returned evidence contributes to the answer, or the capability should be renamed and qualified as tool-enabled question answering.","section":"Section 2.1, Eq. (1), Appendix C.4, Section 3.2"},{"comment":"All accuracy and latency numbers in Table 3, Table 4, Table 5, and Figure 3 come from a single evaluation run, and the paper states that standard deviations and significance tests are not reported. Given that the central claims are quantitative relative improvements (53.7%-228.1% accuracy gains and a 66.2% latency reduction), single-run point estimates provide no indication of variability or robustness. At minimum, repeated runs or bootstrap confidence intervals on a subset such as the 616-question diagnostic set should be provided, or the claims should be softened to single-run observations.","section":"Appendix C.6, C.7"}],"minor_comments":[{"comment":"The column abbreviations in Table 1 (e.g., 'Omni', 'MT', 'Pro.') are not defined in the caption; please spell them out or add a legend.","section":"Table 1 caption"},{"comment":"The judge function Judge(·) is used in Eq. (1) before the judge model (Gemini 3.1 Pro) is introduced in Section 3.1; moving the judge specification to Section 2.1 would improve readability.","section":"Section 2.1, Eq. (1)"},{"comment":"Figure 2 is very dense; labeling the worker roles directly in the figure or adding a short legend for the solid and dashed flows would make the architecture easier to follow.","section":"Figure 2"},{"comment":"The large number of hyperparameters in Appendix C.5 would be easier to digest as a summary table, with the prose paragraphs reserved for details.","section":"Appendix C.5"}],"recommendation":"major_revision","confidential_remarks":"The benchmark artifact and the diagnostic analyses are solid contributions, but the paper's central comparison is weakened by the harness inequality and the Tool-metric validity issue. Both are fixable within the manuscript's scope, so I do not recommend rejection. One additional editorial point: the paper's use of HyperEyes [35] as the crop-based search protocol is a self-citation by overlapping authors, and the dependence of StreamMind's design on that protocol should be stated plainly in the main text if it is retained."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"StreamArena is a real step forward for evaluating always-on streaming video understanding. The dataset—243 hour-scale videos, 3,646 open-ended, timestamped QA pairs covering perception, retrospection, proactive interaction, and tool use—is a genuinely new combination, and the annotation pipeline with dual cross-validation and blind audit is solid. The release of code, data index, and hyperparameters is also a plus. StreamMind's two-tier architecture is a clean design, and the same-backbone latency reduction (66.2%) is a credible systems result.\n\nThe problem is the headline comparative claim. The paper compares StreamMind (continuous ingestion, hidden state) against baselines evaluated through query-triggered replay, which reconstructs a causal prefix or window per question and carries no hidden video state across tasks. That is acknowledged in Appendix C.1, but it is not a level playing field: always-on models like AURA and MiniCPM-o are evaluated as if they were stateless. The 53.7–228.1% improvements over 'strongest streaming baselines' are therefore upper bounds on the real advantage, and the paper should say so in the abstract and conclusion.\n\nThe tool metric is the other soft spot. Section 2.1 explicitly says answer correctness does not establish tool invocation. For Tool questions, many answers (actor parent nationalities, birthdays) are in the backbone's parametric memory, so correct answers with zero tool calls are plausible. The 228.1% Tool improvement may reflect world-knowledge recall, not tool use. This is fixable—require a traceable tool call or post-hoc verification of the search trace—but as reported, one of the four headline capabilities is not actually measured.\n\nA smaller issue: every accuracy number is a single run with no error bars or significance tests. For a benchmark this expensive that is understandable, but it means the architecture comparisons are indicative, not conclusive.\n\nDespite these flaws, the benchmark artifact is valuable. The diagnostics (modality ablation, evidence budgets, reasoning mode) are informative, and the dataset will be useful. I would send this to peer review with major revision, and I would cite the benchmark. The comparative claims need re-scoping or re-verification, but the core contribution stands.","headline":"A genuinely useful hour-scale streaming-video benchmark with a clean reference architecture, but the headline accuracy claims rest on an unequal baseline harness and an unverified tool metric.","tokens_in":20001,"tokens_out":5431,"would_cite":true,"duration_ms":47013,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StreamArena introduces an hour-scale, open-ended benchmark for streaming video understanding, and StreamMind, a two-tier system with persistent asynchronous memory, ranks first among streaming systems on all four capabilities while…","keywords":["streaming video understanding","long-horizon memory","multimodal agents","proactive interaction","benchmark","open-ended evaluation","tool use","persistent memory"],"falsifier":"Run the recent-window baselines in their native continuous-ingestion mode with state persisted across all questions on StreamArena, then compare historical-retrospection and tool accuracy; if those numbers rise to or above StreamMind's reported 34.9% HR and 11.6% Tool, the claimed 53.7%-228.1% improvements are artifacts of the replay harness.","tokens_in":19037,"feed_emoji":"🎬","tokens_out":9038,"duration_ms":84586,"temperature":0.7,"pith_summary":"This paper tries to establish that current streaming-video evaluation is misleading and that a decoupled system design can fix both measurement and performance. It introduces StreamArena, 243 hour-scale videos with 3,646 open-ended, timestamped question-answer pairs covering real-time perception, historical retrospection, proactive alerting, and tool use. Under this test, minimal baselines that only look at the latest few frames, methods that convert past video into text, and methods that repeatedly compress visual memory all fail on at least one capability. The paper's StreamMind separates fast frontend interaction from asynchronous backend memory construction and retrieval, and the reported results improve over the strongest streaming baseline in each capability by 53.7% to 228.1% while reducing pooled query-to-answer latency by 66.2% on a shared backbone. If these results hold, always-on agents can be both more responsive and more accurate by reusing persistent multimodal state built before the question arrives.","feed_headline":"StreamMind tops streaming video baselines and cuts latency 66%","feed_subtitle":"Hour-scale open-ended benchmark shows why recent frames, text summaries, and compressed memory all fail.","key_machinery":"The load-bearing mechanism is the two-tier StreamMind architecture. A frontend tier pairs a Front Worker, which answers, delegates, or spawns monitors, with independently scheduled Monitor Workers that track future conditions without blocking interaction. A backend tier runs a Memory Writer that continuously turns incoming frames and speech into a Memory Bank containing a hierarchical event layer (micro, macro, and super events), an entity-relation graph, and persistent key frames, plus Router, Recall, and Search workers that retrieve and synthesize evidence on demand. Because memory is built before a query arrives and is reused across questions, the latency-critical path shrinks to frontend dispatch, targeted retrieval, and final inference; the same persistent state also preserves multimodal evidence that text summaries or repeated compression lose.","core_discovery":"The paper's central claim is that long-horizon streaming video understanding is best evaluated as a causal, open-ended, multi-capability task rather than as short-clip multiple choice. Its benchmark forces each system to answer without answer choices, with the query timestamp and every supporting evidence segment annotated separately, so a model cannot cheat by recency or language priors. The central finding is a tension: recent-window models miss distant events, text-summary models lose visual evidence, and model-internal compression models lose fine detail over time. StreamMind addresses the tension with two tiers: frontend workers handle immediate interaction and condition monitoring, while backend workers asynchronously build a hierarchical event memory, an entity-relation graph, and key frames, then retrieve on demand. Under a shared 397B-parameter backbone, StreamMind ranks first among streaming systems on all four capabilities, improving on the strongest corresponding baseline by 58.4% on real-time perception, 53.7% on historical retrospection, 228.1% on tool use, and 54.7% on proactive interaction, while reducing pooled query-to-answer latency by 66.2% and retaining 89.7% of pooled accuracy.","pith_inferences":["Editorial extension: the reported gains compare against baseline harnesses that reconstruct a causal prefix or recent window per question and carry no hidden video state; running recent-window systems in true continuous-ingestion mode with state preserved across tasks could narrow the 53.7%-228.1% improvements.","Editorial extension: if the 66.2% latency reduction transfers beyond this benchmark, the same frontend-backend decoupling could apply to embodied or wearable agents where continuous sensing and interactive response compete for the same model.","Editorial extension: ingestion at 2 fps with five-second audio windows makes very short transient events invisible to StreamMind; adaptive sampling or higher capture rates near detected salience might further lift real-time perception.","Editorial extension: the benchmark's 243 videos are heavily Mandarin-dominant (189 of 243), so the reported capability numbers may not fully reflect performance on other languages and cultural contexts."],"forward_implications":["Short-clip, multiple-choice streaming benchmarks can be gamed: a system using only the last four frames can match or beat complex streaming models, so hour-scale open-ended tasks are needed to expose real comprehension.","No single memory design covers all four capabilities; recent windows, text summaries, and internal compression each fail on at least one axis, so continuous interaction and long-horizon comprehension require separate execution paths.","Persistent multimodal memory with retrievable key frames improves historical retrospection at every evidence-to-query gap, not just at short gaps.","Reusing state built before a query reduces query-to-answer latency by 66.2% on a shared backbone while retaining 89.7% of pooled accuracy.","Proactive monitoring needs dedicated workers with a timed alert rule; a turn-based model that only answers when prompted cannot receive credit for autonomous alerting."],"supporting_citations":[{"why":"Documents that a four-frame-only baseline matches or surpasses complex streaming models, motivating the benchmark's open-ended hour-scale design.","marker":"[1]"},{"why":"Supplies the recent-window streaming baseline class whose historical accuracy drops as the evidence-to-query gap grows.","marker":"[13]"},{"why":"Supplies the second recent-window baseline used for comparison on perception, retrospection, tool use, and proactive tasks.","marker":"[20]"},{"why":"Supplies the text-summary baseline class, showing that converting past visual evidence to text loses visual detail.","marker":"[21]"},{"why":"Supplies a model-internal compression baseline with persistent event memory, used to show compression weakens long-horizon recall.","marker":"[22]"},{"why":"Supplies a model-internal compression baseline with low accuracy across all four capabilities, establishing the room for improvement.","marker":"[23]"}],"fun_headline_variants":["StreamMind beats baselines on all four video tasks, cuts latency 66%","StreamArena benchmark shows why recent frames fail; StreamMind wins","StreamMind: 66% faster answers, top scores on hour-long video QA","StreamMind outranks streaming baselines on all four, cuts latency 66%","New benchmark tests hour-scale video memory; StreamMind tops all"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise, stated in Appendix C.1, is that running each streaming baseline through its released query-triggered replay interface—which reconstructs a causal prefix or recent window per task and carries no hidden video state across questions—faithfully represents how that system would perform under continuous ingestion; if always-on models are substantially better in their native mode, the reported 53.7%-228.1% gains would shrink.","fun_headline_variants_meta":{"raw":{"variants":["StreamMind beats baselines on all four video tasks, cuts latency 66%","StreamArena benchmark shows why recent frames fail; StreamMind wins","StreamMind: 66% faster answers, top scores on hour-long video QA","StreamMind outranks streaming baselines on all four, cuts latency 66%","New benchmark tests hour-scale video memory; StreamMind tops all"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000923,"raw_usage":{"total_tokens":3996,"prompt_tokens":1022,"completion_tokens":2974,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":2874}},"tokens_in":638,"tokens_out":2974,"duration_ms":21510,"temperature":1.0,"reasoning_tokens":2874,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:44:49.776340+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the recent-window baselines in their native continuous-ingestion mode with state persisted across all questions on StreamArena, then compare historical-retrospection and tool accuracy; if those numbers rise to or above StreamMind's reported 34.9% HR and 11.6% Tool, the claimed 53.7%-228.1% improvements are artifacts of the replay harness.","supporting_citations":[{"cited_title":"Streamforest: Efficient online video understanding with persistent event memory","cited_arxiv_id":null,"evidence_quote":"Supplies a model-internal compression baseline with persistent event memory, used to show compression weakens long-horizon recall."}],"review_version":1}