{"id":"80ccf765-a923-4d3c-ac4c-4ba83e4191d5","arxiv_id":"2412.03567","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new benchmark and task for detecting the start of a natural-language-described event in streaming egocentric video, with new metrics and adapter-based baselines.","lead":"This paper introduces a new benchmark task, Streaming Detection of Queried Event Start (SDQES): models watch a streaming egocentric video and must detect, with low latency, the start of an event described in natural language. The paper releases a dataset, new metrics, and adapter-based baselines; current models score far below human performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark validity rests on an unvalidated alignment between GPT-4-generated query text and Ego4D annotation start times; the human check in B.1 covers only 87 of 12,767 annotations.","rationale":"The reader's weakest_assumption identifies exactly this issue: the accuracy of Ego4D narrations and temporal boundaries, with only GPT-4 self-checks rather than full human verification. I agree that this is the most load-bearing concern because both the benchmark's value and the empirical conclusions ('all adapter models outperform zero-shot CLIP') rest on the ground-truth starts. The human baseline in B.1 is real evidence and deserves credit, but it is too small and too vaguely described to fully close the gap. I also considered the novelty claim ('no prior work has combined natural language event specification with online prediction') as a potential alternative; however, the paper's related-work discussion is reasonably thorough, and a literature-search objection would be less concrete without a specific counterexample. The benchmark validity concern is more directly testable and is supported by the paper's own Limitations section, which concedes that narrations may lack important details. A conditional verdict remains appropriate: the paper should either provide a larger, well-sampled human validation or explicitly document the reliability limits of the start annotations before the benchmark is widely adopted. Therefore I do not recommend changing the reader's verdict.","tokens_in":156,"tokens_out":5385,"duration_ms":66343,"concrete_test":"Randomly sample 300 EgoSDQES annotations stratified by source (Moments vs NLQ) and by video-length bucket. Have at least three human annotators independently watch the video and mark the start time of the generated query, using the same instructions as in B.1. Compute the proportion of queries for which the human-marked start falls within [ts−5, ts+10] seconds of the dataset ts (human SR@1), as well as the distribution of absolute differences and inter-annotator agreement. If human SR@1 on this sample is substantially below the reported 72.4, or if the median absolute difference exceeds 10 seconds, the ground-truth start alignment is unreliable and the benchmark's metrics and baseline rankings would need to be re-examined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that EgoSDQES enables progress on SDQES depends on the reliability of the ground-truth event start times. The construction pipeline (Section 4, E.1) takes an Ego4D Moments/NLQ annotation, whose interval start becomes es, and asks GPT-4 to generate a query for \"the point at which the event becomes true.\" The LLM sees only narrations and the event label, not the video, so the query can mention preconditions or precursors whose perceived start differs from the annotation interval's start. Section B.3 attempts to quantify ambiguity by measuring annotator variance from overlapping Moments annotations, but this only estimates disagreement among original Ego4D annotators; it does not validate that the generated query text points to the same start. The only direct validation is the human baseline in B.1, evaluated on 87 annotations (0.68% of the dataset), with no description of how those were sampled or whether they cover both Moments and NLQ sources. If for a substantial fraction of queries the human-perceived start differs from es by more than the [−5, +10] second tolerance, then both SR and SMD, and the models trained on these labels, are measuring a misaligned target. This is the load-bearing assumption behind the benchmark's usefulness and the reported improvements over zero-shot CLIP.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Streaming Detection of Queried Event Start (SDQES), a task in which a model must detect, from a streaming egocentric video, the start of an event described by a free-form natural-language query. The authors construct EgoSDQES by adapting Ego4D Moments and NLQ annotations with a GPT-4-based pipeline that generates queries from narrations and event labels, and they propose two metrics, Streaming Recall (SR) and Streaming Minimum Distance (SMD), which respect the online temporal ordering of predictions. They evaluate adapter-based baselines on frozen CLIP, EgoVLP, LaViLa, and EgoVideo backbones, reporting results on 1-minute, 5-minute, and full-length video settings. The main claims are that SDQES/EgoSDQES fill a novel gap at the intersection of online and language-conditioned video understanding, and that the proposed temporal adapters, especially the QR-Adapter, outperform zero-shot CLIP and other adapter variants while remaining computationally efficient.","tokens_in":29583,"tokens_out":5263,"duration_ms":53089,"significance":"If the benchmark's ground truth is valid, the paper makes a useful contribution: the task formulation is a natural and currently underexplored combination of online detection, egocentric video, and open-vocabulary language queries; the SR/SMD metrics improve on p-mAP by accounting for temporal order and false-positive budgets; and the baselines are sensible parameter-efficient starting points. The authors also provide strong reproducibility assets: code for generation and baselines, Weights & Biases logs, Croissant metadata, and a datasheet. However, the central value of the benchmark rests on the assumption that the GPT-4-generated queries share the start time of the source Ego4D annotations, and the empirical comparisons are weakened by threshold selection on the validation split and the absence of error bars. With targeted validation and a cleaner evaluation protocol, the paper could become a solid datasets-and-benchmarks contribution.","major_comments":[{"comment":"The load-bearing assumption that each generated query's semantic start coincides with the source annotation's start time is not adequately validated. The pipeline assigns es from the Ego4D Moments/NLQ interval, while GPT-4 generates the query from narrations and the event label only, without seeing the video; the query may describe a precursor or precondition whose perceived start differs from es. Section B.3 calibrates the [−5, +10] second tolerance using variance among original Ego4D annotators, not agreement between human readers of the generated queries and es. The only direct human check, Section B.1, covers 87 annotations (0.68% of the dataset), with no description of how they were sampled or whether both Moments and NLQ sources are represented. I would like to see a larger, stratified human study in which annotators mark the start of the event described by the generated query, with the distribution of disagreement from es reported separately for Moments and NLQ queries; this is necessary to establish that the benchmark measures the intended target.","section":"Sections 4, E.1, B.1, B.3"},{"comment":"The evaluation protocol can yield optimistically biased results. Section D.1 states that the prediction threshold is selected from 20 candidates by maximizing SR@1 on randomly sampled windows, and the resulting thresholds are then applied to a 'standardized validation set' on the same video split; no separate test split is used. Selecting thresholds on any portion of the same split whose metrics are reported is a form of selection on the evaluation set, and it can inflate absolute scores and distort model comparisons. The absence of error bars, acknowledged in Checklist 3(c), compounds this: several reported differences in Table 2 are small in magnitude. Please restructure the protocol so that threshold selection and final reporting use disjoint subsets (or nested cross-validation), and report metrics with multiple seeds or bootstrap confidence intervals.","section":"Section D.1 and Checklist 3(c)"},{"comment":"The claim that 'every adapter model outperformed the zero-shot CLIP baseline' is too broad. In the 1-minute setting, EgoVLP + ST-Adapter has SR@1 = 17.4 versus 16.9 for zero-shot CLIP but SMD@1 = 30.5 versus 24.3, and EgoVideo + Adapter has SMD@1 = 28.8 versus 24.3; in the 5-minute setting, LaViLa + Adapter, EgoVLP + Adapter, EgoVLP + ST-Adapter, and EgoVLP + RN-Adapter all have worse SMD@1 than zero-shot CLIP. The conclusion should be restricted to the SR metric at the 1-minute clip setting, or the text should explicitly qualify the cases where adapters are not better on SMD.","section":"Section 6.2 and Table 2"}],"minor_comments":[{"comment":"The abstract inconsistently reports 'three vision-language backbones' and 'three adapter architectures' in one passage and 'four vision-language backbones and three adapter architectures' in another; the experiments actually include CLIP, EgoVLP, LaViLa, and EgoVideo as backbones and four adapter variants (Adapter, ST-Adapter, QR-Adapter, RN-Adapter).","section":"Abstract"},{"comment":"There is a duplicate 'Figure 4' caption: one for dataset statistics and another above the ViT-block diagram, which should be renumbered as Figure 5.","section":"Section 5 (Figure captions)"},{"comment":"The phrase 'thresholds established through the randomly sampled windows during training' is confusing, since threshold selection is performed on validation windows rather than during model training; please rewrite to name the three sets explicitly (training windows, threshold-selection windows, final evaluation windows).","section":"Section D.1"},{"comment":"The reported annotator variance is written as sigma^2 = 28.8 seconds; since variance is not measured in seconds, please report the standard deviation and its units clearly.","section":"Section B.3"},{"comment":"Small typos: 'refered to simply as Adapter' should be 'referred', and 'initalization' in C.5.1 should be 'initialization'.","section":"Sections 6 and C.5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper targets a real gap and provides valuable public assets, but the benchmark's validity depends on the alignment between LLM-generated queries and inherited Ego4D start times, which is currently supported only by a very small human check and an indirect calibration. The threshold-selection protocol also needs to be cleaned up. I would be willing to support acceptance after these points are addressed with additional validation and a clearer evaluation split."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces Streaming Detection of Queried Event Start (SDQES), a task that sits in a real gap: online detection of natural-language-defined event starts in egocentric video. The EgoSDQES benchmark, built from Ego4D, is a practical contribution, and the SR/SMD metrics are thoughtfully designed to handle ambiguity and false positives. The supplementary material is genuinely thorough—datasheet, code, data, W&B logs, human baseline, and a discussion of limitations. That is more than many benchmark papers ship.\n\nThe soft spots are real but mostly fixable. No error bars (the authors admit this and say they will add them), and the threshold-selection protocol in D.1 is worded in a way that makes it hard to tell whether the validation set is being used twice. The human verification covers only 87 of 12,767 annotations; that is acknowledged, but the sampling procedure is not described. There are also two small internal inconsistencies (abstract says three backbones, text says four; D.1 says EgoVideo is limited to 30 frames, Table 5 says 50). None of these sink the central claim.\n\nThe stress-test note worries that GPT-4-generated queries may not point at the same start time as the original Ego4D annotation. That is a legitimate concern, but I think it is overstated. The pipeline starts from the original annotation's event label and narrations, and the LLM is explicitly asked to generate a query for when that event becomes true. The human baseline, small as it is, shows 72.4% SR@1, which is direct evidence that humans can match the annotation starts for a sample of queries. The tolerance window of [−5,+10] seconds is also anchored to annotator variance. Still, the authors should expand the human check and describe how the 87 were sampled; the current number is thin for a benchmark that others will train on.\n\nThe reader's circularity score of 2 seems right—the threshold selection is standard model selection, not metric fitting. The soundness score of 4 is a bit harsh; the evaluation is not broken, just underreported.\n\nBottom line: this is a useful benchmark paper for anyone working on online video-language understanding or egocentric assistant systems. It deserves serious peer review and, after the evaluation details are tightened, acceptance. I would take it to reading group.","headline":"A genuinely new benchmark task with a thorough release, but the evaluation needs error bars, a clearer threshold protocol, and a bigger human check before it can anchor follow-up work.","tokens_in":30098,"tokens_out":3148,"would_cite":true,"duration_ms":30395,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SDQES proposes that a streaming system should announce the start of a natural-language-described event in egocentric video, and supplies a benchmark, two metrics, and adapter baselines to make that measurable.","keywords":["streaming video understanding","natural language event detection","event start detection","egocentric video","online action detection","vision-language models","parameter-efficient adapters","video-language benchmark"],"falsifier":"Take a random sample of EgoSDQES queries, show independent annotators each video without revealing the generated start time, and ask them to mark when the queried event begins; if a large fraction of human marks fall outside the metric's $[-5, +10]$ second tolerance of the benchmark start, the ground truth does not support the reported measurements.","tokens_in":29109,"feed_emoji":"⏱️","tokens_out":9484,"duration_ms":87303,"temperature":0.7,"pith_summary":"The paper proposes Streaming Detection of Queried Event Start (SDQES): given live egocentric video and a natural-language query such as \"remind me to wash my hands when I throw away the trash,\" a model must signal the moment the described event begins, before seeing the future. The authors argue that existing online action detection works from a fixed class list and that offline language-based localization cannot see the future, leaving this combination unstudied. To make the task concrete, they build EgoSDQES from Ego4D by having an LLM convert temporally grounded annotations and narrations into streaming queries, yielding 12,767 queries over 1,773 untrimmed videos. They introduce Streaming Recall (correct if one of the first k predictions lands within a tolerance window of the true start) and Streaming Minimum Distance, and evaluate frozen vision-language backbones equipped with lightweight temporal adapters. If the benchmark's ground truth holds, SDQES gives embodied applications a measurable target for low-latency, open-vocabulary event detection.","feed_headline":"Streaming task detects event starts in egocentric video as they happen","feed_subtitle":"A new benchmark pairs 12,767 natural-language queries with egocentric videos so models can flag the exact start moment.","key_machinery":"The load-bearing mechanism is the benchmark construction pipeline combined with the streaming evaluation metric. Starting from Ego4D Moments and NLQ annotations plus dense narrations, the paper prompts GPT-4 to extract the event, check that it is grounded in narrations, check whether it occurred earlier in the video, and rewrite it as a first-person streaming reminder query; a specificity filter keeps only queries that point to the intended instance without future context. On the evaluation side, Streaming Recall declares a model's output correct if any of its first k predictions falls within the asymmetric window $[-5, +10]$ seconds around the annotated start, a window chosen from observed annotator variance in Ego4D. The model family carries the online requirement: frozen vision-language encoders (CLIP, EgoVLP, LaViLa, EgoVideo) with small temporal adapters, such as 1D convolutions, QRNN, or RetNet, that update incrementally per frame rather than reprocessing the whole history.","core_discovery":"The paper's central claim is that Streaming Detection of Queried Event Start (SDQES) is a coherent, previously unstudied task: given an untrimmed egocentric video stream and a natural-language query, a model must output the start time of the described event while seeing only past frames, and no prior work combined open-vocabulary language queries with online prediction. To make the task concrete, the paper constructs EgoSDQES from Ego4D by converting existing temporally grounded annotations and narrations into 12,767 streaming queries across 1,773 videos, and proposes two metrics: Streaming Recall, which accepts a prediction if one of the model's first k outputs falls in an asymmetric tolerance window around the true start, and Streaming Minimum Distance, which measures how close the best of the first k predictions gets. On this benchmark, frozen image-language backbones equipped with lightweight temporal adapters all outperform a zero-shot CLIP baseline, and QRNN-based temporal aggregation gives the largest gains, indicating that temporal modeling matters for the task. The paper presents these as initial baselines rather than as a solved problem.","pith_inferences":["One direction the paper leaves open is whether models trained on EgoSDQES detect starts of fresh human-authored queries, which is the actual deployment condition for an assistant.","The same LLM conversion pipeline could be applied to other temporally grounded video corpora beyond the datasets it was tested on, effectively minting new streaming-query benchmarks from existing annotations.","Streaming Recall's first-k rule rewards conservative models that delay predictions, while Streaming Minimum Distance rewards closeness once a prediction is made; a single combined score might give a truer picture of deployment readiness.","A testable extension would be to split results by query type, such as events with clear visual precursors versus events requiring contextual reasoning, to see where observation latency is concentrated."],"forward_implications":["A system that solves SDQES can announce \"now\" at the moment a requested event begins in first-person video, which is the trigger condition for assistive, robotic, and driving applications.","EgoSDQES gives the field 12,767 natural-language start-detection queries over 1,773 untrimmed egocentric videos, enabling training and evaluation on a common ground.","Streaming Recall@k penalizes false positives directly: a model that fires too many early alarms exhausts its k predictions before the true start, so accuracy and precision are measured jointly.","Lightweight temporal adapters on frozen image-language backbones reach the streaming setting with roughly constant per-frame cost, unlike sliding-window processing that multiplies computation by about four.","Open-vocabulary queries remove the fixed action-list constraint of earlier online detection, so users can specify arbitrary events rather than choosing from a predefined taxonomy."],"supporting_citations":[{"why":"Ego4D supplies the videos, narrations, Moments annotations, and NLQ annotations that EgoSDQES is built from.","marker":"[3]"},{"why":"Defines online detection of action start, the closed-vocabulary predecessor that SDQES extends to natural-language queries.","marker":"[9]"},{"why":"CLIP is the zero-shot baseline and the frozen image-language backbone on which the adapter models are trained.","marker":"[52]"},{"why":"ST-Adapter is the image-to-video transfer method whose temporal-adapter design the streaming baselines build on.","marker":"[54]"},{"why":"GPT-4 performs the query generation, grounding check, prior-occurrence check, and specificity filtering.","marker":"[61]"},{"why":"EgoVLP is one of the egocentric vision-language backbones evaluated with adapters.","marker":"[12]"},{"why":"LaViLa is another egocentric vision-language backbone evaluated with adapters.","marker":"[46]"},{"why":"EgoVideo is the state-of-the-art egocentric dual-encoder backbone evaluated with adapters.","marker":"[47]"},{"why":"EgoExoLearn is used to show the generation pipeline extends to another egocentric dataset.","marker":"[60]"},{"why":"QRNN supplies the recurrent architecture for the QR-Adapter, the best-performing temporal adapter in the experiments.","marker":"[64]"}],"fun_headline_variants":["New task: spot event start in live egocentric video via language queries","Streaming query-event start detection benchmark built from Ego4D","Model flags exact start moment of queried events in video streams","Low-latency detection of event beginnings from natural language cues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is only as reliable as the Ego4D narrations and temporal annotations it is built from, plus the ability of GPT-4 to turn them into unambiguous queries; incomplete narrations or misaligned start times would make both training and the reported metrics rest on faulty labels.","fun_headline_variants_meta":{"raw":{"variants":["New task: spot event start in live egocentric video via language queries","Streaming query-event start detection benchmark built from Ego4D","Model flags exact start moment of queried events in video streams","Low-latency detection of event beginnings from natural language cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1374,"prompt_tokens":920,"completion_tokens":454,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":536,"tokens_out":454,"duration_ms":4736,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:15:04.542372+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of EgoSDQES queries, show independent annotators each video without revealing the generated start time, and ask them to mark when the queried event begins; if a large fraction of human marks fall outside the metric's $[-5, +10]$ second tolerance of the benchmark start, the ground truth does not support the reported measurements.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ego4D supplies the videos, narrations, Moments annotations, and NLQ annotations that EgoSDQES is built from."},{"cited_title":"Online detection of action start in untrimmed, streaming videos","cited_arxiv_id":null,"evidence_quote":"Defines online detection of action start, the closed-vocabulary predecessor that SDQES extends to natural-language queries."},{"cited_title":"St-adapter: Parameter- efficient image-to-video transfer learning, 2022","cited_arxiv_id":null,"evidence_quote":"ST-Adapter is the image-to-video transfer method whose temporal-adapter design the streaming baselines build on."},{"cited_title":"Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world","cited_arxiv_id":null,"evidence_quote":"EgoExoLearn is used to show the generation pipeline extends to another egocentric dataset."}],"review_version":1}