Pith. sign in

REVIEW 3 major objections 5 minor 77 references

Streaming Detection of Queried Event Start

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SDQES proposes that a streaming system should announce the start of a natural-language-described event in egocentric video, and supplies a benchmark, two metrics, and adapter baselines to make that measurable.

desk verdict A genuinely new benchmark task with a thorough release, but the evaluation needs error bars, a clearer threshold protocol, and a bigger human check before it can anchor follow-up work. read the letter →

arxiv 2412.03567 v1 pith:GDDCJBHD submitted 2024-12-04 cs.CV

Cristobal Eyzaguirre , Eric Tang , Shyamal Buch , Adrien Gaidon , Jiajun Wu , Juan Carlos Niebles This is my paper · ORCID
classification cs.CV
keywords streamingvideounderstandingnaturallanguageeventdetectionstartegocentriconlineactionvision-languagemodelsparameter-efficientadaptersvideo-languagebenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Streaming Detection of Queried Event Start (SDQES): given live egocentric video and a natural-language query such as "remind me to wash my hands when I throw away the trash," a model must signal the moment the described event begins, before seeing the future. The authors argue that existing online action detection works from a fixed class list and that offline language-based localization cannot see the future, leaving this combination unstudied. To make the task concrete, they build EgoSDQES from Ego4D by having an LLM convert temporally grounded annotations and narrations into streaming queries, yielding 12,767 queries over 1,773 untrimmed videos. They introduce Streaming Recall (correct if one of the first k predictions lands within a tolerance window of the true start) and Streaming Minimum Distance, and evaluate frozen vision-language backbones equipped with lightweight temporal adapters. If the benchmark's ground truth holds, SDQES gives embodied applications a measurable target for low-latency, open-vocabulary event detection.

What carries the argument

The load-bearing mechanism is the benchmark construction pipeline combined with the streaming evaluation metric. Starting from Ego4D Moments and NLQ annotations plus dense narrations, the paper prompts GPT-4 to extract the event, check that it is grounded in narrations, check whether it occurred earlier in the video, and rewrite it as a first-person streaming reminder query; a specificity filter keeps only queries that point to the intended instance without future context. On the evaluation side, Streaming Recall declares a model's output correct if any of its first k predictions falls within the asymmetric window $[-5, +10]$ seconds around the annotated start, a window chosen from observed annotator variance in Ego4D. The model family carries the online requirement: frozen vision-language encoders (CLIP, EgoVLP, LaViLa, EgoVideo) with small temporal adapters, such as 1D convolutions, QRNN, or RetNet, that update incrementally per frame rather than reprocessing the whole history.

What would settle it

Take a random sample of EgoSDQES queries, show independent annotators each video without revealing the generated start time, and ask them to mark when the queried event begins; if a large fraction of human marks fall outside the metric's $[-5, +10]$ second tolerance of the benchmark start, the ground truth does not support the reported measurements.

Watch

Extended reading notes

Core claim

The paper's central claim is that Streaming Detection of Queried Event Start (SDQES) is a coherent, previously unstudied task: given an untrimmed egocentric video stream and a natural-language query, a model must output the start time of the described event while seeing only past frames, and no prior work combined open-vocabulary language queries with online prediction. To make the task concrete, the paper constructs EgoSDQES from Ego4D by converting existing temporally grounded annotations and narrations into 12,767 streaming queries across 1,773 videos, and proposes two metrics: Streaming Recall, which accepts a prediction if one of the model's first k outputs falls in an asymmetric tolerance window around the true start, and Streaming Minimum Distance, which measures how close the best of the first k predictions gets. On this benchmark, frozen image-language backbones equipped with lightweight temporal adapters all outperform a zero-shot CLIP baseline, and QRNN-based temporal aggregation gives the largest gains, indicating that temporal modeling matters for the task. The paper presents these as initial baselines rather than as a solved problem.

Load-bearing premise

The benchmark's ground truth is only as reliable as the Ego4D narrations and temporal annotations it is built from, plus the ability of GPT-4 to turn them into unambiguous queries; incomplete narrations or misaligned start times would make both training and the reported metrics rest on faulty labels.

Editorial extensions

If this is right

  • A system that solves SDQES can announce "now" at the moment a requested event begins in first-person video, which is the trigger condition for assistive, robotic, and driving applications.
  • EgoSDQES gives the field 12,767 natural-language start-detection queries over 1,773 untrimmed egocentric videos, enabling training and evaluation on a common ground.
  • Streaming Recall@k penalizes false positives directly: a model that fires too many early alarms exhausts its k predictions before the true start, so accuracy and precision are measured jointly.
  • Lightweight temporal adapters on frozen image-language backbones reach the streaming setting with roughly constant per-frame cost, unlike sliding-window processing that multiplies computation by about four.
  • Open-vocabulary queries remove the fixed action-list constraint of earlier online detection, so users can specify arbitrary events rather than choosing from a predefined taxonomy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One direction the paper leaves open is whether models trained on EgoSDQES detect starts of fresh human-authored queries, which is the actual deployment condition for an assistant.
  • The same LLM conversion pipeline could be applied to other temporally grounded video corpora beyond the datasets it was tested on, effectively minting new streaming-query benchmarks from existing annotations.
  • Streaming Recall's first-k rule rewards conservative models that delay predictions, while Streaming Minimum Distance rewards closeness once a prediction is made; a single combined score might give a truer picture of deployment readiness.
  • A testable extension would be to split results by query type, such as events with clear visual precursors versus events requiring contextual reasoning, to see where observation latency is concentrated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Streaming Detection of Queried Event Start (SDQES), a task in which a model must detect, from a streaming egocentric video, the start of an event described by a free-form natural-language query. The authors construct EgoSDQES by adapting Ego4D Moments and NLQ annotations with a GPT-4-based pipeline that generates queries from narrations and event labels, and they propose two metrics, Streaming Recall (SR) and Streaming Minimum Distance (SMD), which respect the online temporal ordering of predictions. They evaluate adapter-based baselines on frozen CLIP, EgoVLP, LaViLa, and EgoVideo backbones, reporting results on 1-minute, 5-minute, and full-length video settings. The main claims are that SDQES/EgoSDQES fill a novel gap at the intersection of online and language-conditioned video understanding, and that the proposed temporal adapters, especially the QR-Adapter, outperform zero-shot CLIP and other adapter variants while remaining computationally efficient.

Significance. If the benchmark's ground truth is valid, the paper makes a useful contribution: the task formulation is a natural and currently underexplored combination of online detection, egocentric video, and open-vocabulary language queries; the SR/SMD metrics improve on p-mAP by accounting for temporal order and false-positive budgets; and the baselines are sensible parameter-efficient starting points. The authors also provide strong reproducibility assets: code for generation and baselines, Weights & Biases logs, Croissant metadata, and a datasheet. However, the central value of the benchmark rests on the assumption that the GPT-4-generated queries share the start time of the source Ego4D annotations, and the empirical comparisons are weakened by threshold selection on the validation split and the absence of error bars. With targeted validation and a cleaner evaluation protocol, the paper could become a solid datasets-and-benchmarks contribution.

major comments (3)
  1. [Sections 4, E.1, B.1, B.3] The load-bearing assumption that each generated query's semantic start coincides with the source annotation's start time is not adequately validated. The pipeline assigns es from the Ego4D Moments/NLQ interval, while GPT-4 generates the query from narrations and the event label only, without seeing the video; the query may describe a precursor or precondition whose perceived start differs from es. Section B.3 calibrates the [−5, +10] second tolerance using variance among original Ego4D annotators, not agreement between human readers of the generated queries and es. The only direct human check, Section B.1, covers 87 annotations (0.68% of the dataset), with no description of how they were sampled or whether both Moments and NLQ sources are represented. I would like to see a larger, stratified human study in which annotators mark the start of the event described by the generated query, with the distribution of disagreement from es reported separately for Moments and NLQ queries; this is necessary to establish that the benchmark measures the intended target.
  2. [Section D.1 and Checklist 3(c)] The evaluation protocol can yield optimistically biased results. Section D.1 states that the prediction threshold is selected from 20 candidates by maximizing SR@1 on randomly sampled windows, and the resulting thresholds are then applied to a 'standardized validation set' on the same video split; no separate test split is used. Selecting thresholds on any portion of the same split whose metrics are reported is a form of selection on the evaluation set, and it can inflate absolute scores and distort model comparisons. The absence of error bars, acknowledged in Checklist 3(c), compounds this: several reported differences in Table 2 are small in magnitude. Please restructure the protocol so that threshold selection and final reporting use disjoint subsets (or nested cross-validation), and report metrics with multiple seeds or bootstrap confidence intervals.
  3. [Section 6.2 and Table 2] The claim that 'every adapter model outperformed the zero-shot CLIP baseline' is too broad. In the 1-minute setting, EgoVLP + ST-Adapter has SR@1 = 17.4 versus 16.9 for zero-shot CLIP but SMD@1 = 30.5 versus 24.3, and EgoVideo + Adapter has SMD@1 = 28.8 versus 24.3; in the 5-minute setting, LaViLa + Adapter, EgoVLP + Adapter, EgoVLP + ST-Adapter, and EgoVLP + RN-Adapter all have worse SMD@1 than zero-shot CLIP. The conclusion should be restricted to the SR metric at the 1-minute clip setting, or the text should explicitly qualify the cases where adapters are not better on SMD.
minor comments (5)
  1. [Abstract] The abstract inconsistently reports 'three vision-language backbones' and 'three adapter architectures' in one passage and 'four vision-language backbones and three adapter architectures' in another; the experiments actually include CLIP, EgoVLP, LaViLa, and EgoVideo as backbones and four adapter variants (Adapter, ST-Adapter, QR-Adapter, RN-Adapter).
  2. [Section 5 (Figure captions)] There is a duplicate 'Figure 4' caption: one for dataset statistics and another above the ViT-block diagram, which should be renumbered as Figure 5.
  3. [Section D.1] The phrase 'thresholds established through the randomly sampled windows during training' is confusing, since threshold selection is performed on validation windows rather than during model training; please rewrite to name the three sets explicitly (training windows, threshold-selection windows, final evaluation windows).
  4. [Section B.3] The reported annotator variance is written as sigma^2 = 28.8 seconds; since variance is not measured in seconds, please report the standard deviation and its units clearly.
  5. [Sections 6 and C.5.1] Small typos: 'refered to simply as Adapter' should be 'referred', and 'initalization' in C.5.1 should be 'initialization'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark labels come from external Ego4D annotations, the metrics are not fitted to model outputs, and the central claims are evaluated on held-out splits rather than by self-referential construction.

full rationale

The paper's derivation chain is self-contained against external data and benchmarks, so no circular step meets the evidentiary bar. The SDQES task definition and EgoSDQES benchmark are built by repurposing Ego4D Moments/NLQ annotations: the ground-truth event start times are the start times of existing Ego4D temporal annotations, and GPT-4 generates queries from narrations and event labels (Sections 4, E.1). These labels are external inputs, not outputs of the proposed models, so training and evaluation do not predict a fitted value by construction. The metrics Streaming Recall and Streaming Minimum Distance are defined in Section 3.2 directly from the task formulation, and the asymmetric tolerance window [−5, +10] is chosen in Section B.3 from the variance of Ego4D annotator start times (σ = 5.3 seconds), not from model performance; this is a data-driven evaluation choice rather than a fitted parameter disguised as a prediction. Threshold selection in Section D.1 is a standard validation-based model-selection step: thresholds are chosen on randomly sampled windows to maximize SR@1 and then applied to a standardized validation set; even if the protocol is under-specified, no equation makes the reported results equivalent to the selected thresholds. The human baseline in Section B.1 (SR@1 = 72.4 on 87 annotations) is an external verification of annotation quality, not a derivation of the benchmark's claims. Self-citations (e.g., refs. 1, 16, 34, 39) appear only in related-work or background contexts and are not load-bearing; there is no imported uniqueness theorem and no ansatz smuggled in via a self-citation. The Limitations section honestly concedes that Ego4D narrations 'may lack important details' and that LLM-based generation can produce low-quality outputs; those are data-quality and validity risks, not circularity. Under the rule that 'this is not standard consensus' and 'weak assumptions' are correctness concerns rather than circularity, the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the EgoSDQES dataset and the SR/SMD metrics are constructed artifacts rather than postulated entities requiring independent evidence.

free parameters (4)
  • SR tolerance window (anticipation, latency) = -5 to +10 seconds
    Set to one and two standard deviations of annotator start-time variance (sigma=5.3s, Section B.3); values are data-derived rather than model-fitted, but they define the correctness criterion for all reported SR numbers.
  • Prediction threshold = One of 20 candidate values maximizing SR@1
    Chosen on validation windows (Section D.1); the exact protocol is ambiguous and may involve threshold selection on the same split used for evaluation.
  • Training window size ws = 60 frames (30 for RetNet and EgoVideo in main text; Table 5 lists 50 for EgoVideo)
    Hyperparameter set for training stability; the discrepancy between 30 and 50 for EgoVideo is unresolved.
  • Adapter bottleneck dimension d' = Not stated in paper; chosen to roughly equalize adapter parameter counts
    Section D.2 states d' is adjusted so adapters have roughly equal parameters, making comparisons fair but leaving the exact value unspecified.
assumptions (4)
  • domain assumption Ego4D narrations and temporal annotations are sufficiently accurate and complete to serve as ground truth for LLM-generated streaming queries.
    Sections 4 and E.1 rely on narrations for event grounding; the paper itself notes in Limitations that narrations 'may lack important details, violating assumptions about their quality.'
  • domain assumption GPT-4 can reliably extract events, generate specific streaming queries, and self-check grounding and specificity.
    The entire dataset construction pipeline (Section 4, E.1, E.2) uses GPT-4 without human verification on the full set; only 87 annotations were human-checked in the supplementary material.
  • domain assumption Event start boundaries from Ego4D Moments/NLQ correspond to the starts of the generated queries.
    The paper reuses original temporal annotations as ground truth for the new queries; ambiguity in start boundaries is acknowledged in Limitations and mitigated only by the tolerance window.
  • ad hoc to paper A tolerance window of [-5, +10] seconds and the first-k predictions criterion adequately capture streaming detection quality.
    The window is derived from annotator variance (Section B.3), and the first-k rule is a design choice that penalizes false positives; both are reasonable but not independently validated against an external benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Streaming Detection of Queried Event Start." pith.science (2026). https://pith.science/paper/GDDCJBHD

@misc{pith2026241203567,
  author       = {Pith},
  title        = {Pith review of: Streaming Detection of Queried Event Start},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDDCJBHD}},
  note         = {Machine review of arXiv:2412.03567}
}
read the original abstract

Robotics, autonomous driving, augmented reality, and many embodied computer vision applications must quickly react to user-defined events unfolding in real time. We address this setting by proposing a novel task for multimodal video understanding-Streaming Detection of Queried Event Start (SDQES). The goal of SDQES is to identify the beginning of a complex event as described by a natural language query, with high accuracy and low latency. We introduce a new benchmark based on the Ego4D dataset, as well as new task-specific metrics to study streaming multimodal detection of diverse events in an egocentric video setting. Inspired by parameter-efficient fine-tuning methods in NLP and for video tasks, we propose adapter-based baselines that enable image-to-video transfer learning, allowing for efficient online video modeling. We evaluate three vision-language backbones and three adapter architectures on both short-clip and untrimmed video settings.

Figures

Figures reproduced from arXiv: 2412.03567 by the authors.

Figure 1
Figure 1. Overview of our proposed SDQES task. The goal of streaming detection of queried event start (SDQES) is for a system to detect the start of a complex event, described by natural language, with low latency from a streaming video input. This task is a novel intersection of multimodal event and online/streaming video understanding benchmarks. It is intended to encourage the design of new streaming multimodal models for … view at source ↗
Figure 2
Figure 2. Example videos and queries from our dataset EgoSDQES. 2 Related Work Our proposed streaming detection of queried event start (SDQES) task is a unique intersection of video understanding areas that have not been explored by prior work, which we summarize below. Action Recognition and Detection in Videos. The goal of action recognition systems is to output the action or activity present in an input video clip [14, 15]… view at source ↗
Figure 3
Figure 3. Dataset generation pipeline. Left: we show the generation pipeline steps for an example video with dense captions. Right: Sankey diagram illustrates the flow of data from Ego4D through the various filtering stages. Asterisk (∗) encodes a filter based on query specificity. Streaming Minimum Distance. In addition to Streaming Recall, we propose a second metric that focuses on timeliness: Streaming Minimum Distance (SM… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Dataset statistics. Left: Event duration in seconds. Center: Distribution of Event Start with respect to video start. Right: Word Cloud of the query generations. streaming query for the current instance, following the template of setting a reminder to do something when…
Figure 4
Figure 4. Figure 4: (a) ViT Block with Adapter … … … … W H … … Feature Patches State Temporal Aggregation t … … (b) Adapter Internals [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Full length video results. 6.2 Task Performance Results In our experimental evaluation, we present findings on the performance of various adapter models integrated with dual-encoder vision-language architectures, using video clips of 1-minute and 5- minute duration cap…
Figure 7
Figure 7. Figure 7: Metric Comparison. These two scenarios are indistinguishable for the p-mAP metric. Top: model correctly alerts the user on the third attempt after two false positives. Bottom: model is successful on its first attempt and alert can be turned off, avoiding two future fal…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 48 canonical work pages

  1. [1]

    Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments

    Sanjana Srivastava*, Chengshu Li*, Michael Lingelbach*, Roberto Martín-Martín*, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference in Robot Learning (CoRL), ...

    work page 2021

  2. [2]

    Vision meets robotics: The kitti dataset

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013

    2013

  3. [3]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagara- jan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car...

    work page 2022

  4. [4]

    Learning spa- tiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489–4497, 2015

    2015

  5. [5]

    A closer look at spatiotemporal convolutions for action recognition

    Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018

    2018

  6. [6]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017

    2017

  7. [7]

    Red: Reinforced encoder-decoder networks for action anticipation

    Jiyang Gao, Zhenheng Yang, and Ram Nevatia. Red: Reinforced encoder-decoder networks for action anticipation. arXiv preprint arXiv:1707.04818, 2017

    arXiv 2017

  8. [8]

    Temporal action detection with structured segment networks

    Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2914–2923, 2017

    work page 2017

Show all 77 references
  1. [9]

    Online detection of action start in untrimmed, streaming videos

    Zheng Shou, Junting Pan, Jonathan Chan, Kazuyuki Miyazawa, Hassan Mansour, Anthony Vetro, Xavier Giro-i Nieto, and Shih-Fu Chang. Online detection of action start in untrimmed, streaming videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 534–551, 2018

    2018

  2. [10]

    Startnet: Online detection of action start in untrimmed videos

    Mingfei Gao, Mingze Xu, Larry S Davis, Richard Socher, and Caiming Xiong. Startnet: Online detection of action start in untrimmed videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5542–5551, 2019

    2019

  3. [11]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 5267–5275, 2017

    2017

  4. [12]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhong- cong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang, 11 Dima Damen, Bernard Ghanem, Wei Liu, and Mike Zheng Shou. Egocentric video-language pretraining. a...

    2022 arXiv

  5. [13]

    Span-based localizing network for natural language video localization

    Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6543–6554, 2020

    2020

  6. [14]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  7. [15]

    Kuehne, H

    H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In 2011 International Conference on Computer Vision, pages 2556–2563, 2011

    2011

  8. [16]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015

    2015

  9. [17]

    in the wild

    Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumos challenge on action recognition for videos “in the wild”. Computer Vision and Image Understanding, 155:1–23, 2017

    2017

  10. [18]

    Rethinking the faster r-cnn architecture for temporal action localization

    Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar. Rethinking the faster r-cnn architecture for temporal action localization. In proceedings of the IEEE conference on computer vision and pattern recognition , pages 1130–1139, 2018

    2018

  11. [19]

    Temporal action localization in untrimmed videos via multi-stage cnns

    Zheng Shou, Dongang Wang, and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1049–1058, 2016

    2016

  12. [20]

    X3d: Expanding architectures for efficient video recognition

    Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203–213, 2020

    2020

  13. [21]

    R-c3d: Region convolutional 3d network for temporal activity detection

    Huijuan Xu, Abir Das, and Kate Saenko. R-c3d: Region convolutional 3d network for temporal activity detection. In Proceedings of the IEEE international conference on computer vision, pages 5783–5792, 2017

    2017

  14. [22]

    Max-margin early event detectors

    Minh Hoai and Fernando De la Torre. Max-margin early event detectors. International Journal of Computer Vision, 107(2):191–202, 2014

    2014

  15. [23]

    Learning activity progression in lstms for activity detection and early detection

    Shugao Ma, Leonid Sigal, and Stan Sclaroff. Learning activity progression in lstms for activity detection and early detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1942–1950, 2016

    1942

  16. [24]

    M. S. Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In 2011 International Conference on Computer Vision, pages 1036–1043, 2011

    2011

  17. [25]

    Temporal recurrent networks for online action detection

    Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S Davis, and David J Crandall. Temporal recurrent networks for online action detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5532–5541, 2019

    2019

  18. [26]

    Activity forecasting

    Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part IV 12, pages 201–214. Springer, 2012

    2012

  19. [27]

    First-person activity forecasting with online inverse reinforcement learning

    Nicholas Rhinehart and Kris M Kitani. First-person activity forecasting with online inverse reinforcement learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 3696–3705, 2017

    2017

  20. [28]

    Rolling-unrolling lstms for action anticipation from first-person video

    Antonino Furnari and Giovanni Maria Farinella. Rolling-unrolling lstms for action anticipation from first-person video. IEEE transactions on pattern analysis and machine intelligence , 43(11):4021–4036, 2020

    2020

  21. [29]

    Ego-topo: Environment affordances from egocentric video

    Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 163–172, 2020

    2020

  22. [30]

    Anticipative video transformer

    Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13505–13515, 2021. 12

    2021

  23. [31]

    Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition

    Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2022

  24. [32]

    End-to-end concept word detection for video captioning, retrieval, and question answering

    Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. End-to-end concept word detection for video captioning, retrieval, and question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3165–3173, 2017

    2017

  25. [33]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016

    2016

  26. [34]

    Dense- captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In International Conference on Computer Vision (ICCV), 2017

    2017

  27. [35]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE international conference on computer vision, pages 2630...

    2019

  28. [36]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018

    2018

  29. [37]

    End-to-end learning of visual representations from uncurated instructional videos

    Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9...

    2020

  30. [38]

    Merlot: Multimodal neural script knowledge models

    Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. NeurIPS, 2021

    2021

  31. [39]

    Revisiting the “Video” in Video-Language Understanding

    Shyamal Buch, Cristobal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the “Video” in Video-Language Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  32. [40]

    Localizing moments in video with temporal language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with temporal language. In Empirical Methods in Natural Language Processing (EMNLP), 2018

    2018

  33. [41]

    What is more likely to happen next? video-and-language future event prediction

    Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. What is more likely to happen next? video-and-language future event prediction. In EMNLP, 2020

    2020

  34. [42]

    Visu- alcomet: Reasoning about the dynamic context of a still image

    Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visu- alcomet: Reasoning about the dynamic context of a still image. In European Conference on Computer Vision, pages 508–524. Springer, 2020

    2020

  35. [43]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Visio...

    2018

  36. [44]

    The epic-kitchens dataset: Collection, challenges and baselines

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern ...

    2021

  37. [45]

    Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evan- gelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. Internati...

    2022

  38. [46]

    Learning video representations from large language models

    Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In CVPR, 2023

    2023

  39. [47]

    Egovideo: Exploring egocentric foundation model and downstream adaptation

    Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocentric foundation model and downstream adaptation. arXiv preprint arXiv:2406.18070, 2024. 13

    2024 arXiv

  40. [48]

    Groundnlq@ ego4d natural language queries challenge 2023

    Zhijian Hou, Lei Ji, Difei Gao, Wanjun Zhong, Kun Yan, Chao Li, Wing-Kwong Chan, Chong- Wah Ngo, Nan Duan, and Mike Zheng Shou. Groundnlq@ ego4d natural language queries challenge 2023. arXiv preprint arXiv:2306.15255, 2023

    2023 arXiv

  41. [49]

    Naq: Leveraging narrations as queries to supervise episodic memory

    Santhosh Kumar Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Naq: Leveraging narrations as queries to supervise episodic memory. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6694–6703, 2023

    2023

  42. [50]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019

    2019

  43. [51]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021

    2021

  44. [52]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021

    2021 arXiv

  45. [53]

    Adaptformer: Adapting vision transformers for scalable visual recognition

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205.13535, 2022

    2022 arXiv

  46. [54]

    St-adapter: Parameter- efficient image-to-video transfer learning, 2022

    Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter- efficient image-to-video transfer learning, 2022

    2022

  47. [55]

    Frozen clip models are efficient video learners

    Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference on Computer Vision, pages 388–404. Springer, 2022

    2022

  48. [56]

    Aim: Adapting image models for efficient video understanding

    Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video understanding. In International Conference on Learning Representations, 2023

    2023

  49. [57]

    Disentangling spatial and temporal learning for efficient image-to-video transfer learning

    Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Yingya Zhang, Changxin Gao, Deli Zhao, and Nong Sang. Disentangling spatial and temporal learning for efficient image-to-video transfer learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13934–13944, 2023

    2023

  50. [58]

    Side4video: Spatial-temporal side network for memory-efficient image-to-video transfer learning, 2023

    Huanjin Yao, Wenhao Wu, and Zhiheng Li. Side4video: Spatial-temporal side network for memory-efficient image-to-video transfer learning, 2023

    2023

  51. [59]

    Time-, memory- and parameter-efficient visual adaptation

    Otniel-Bogdan Mercea, Alexey Gritsenko, Cordelia Schmid, and Anurag Arnab. Time-, memory- and parameter-efficient visual adaptation. arXiv preprint arXiv:2402.02887, 2024

    2024 arXiv

  52. [60]

    Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world

    Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, et al. Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. InProceedings of the IEEE/CVF Confere...

    2024

  53. [61]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  54. [62]

    Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021

    2021

  55. [63]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In International Conference on Computer Vision (ICCV), 2021

    2021

  56. [64]

    Quasi-recurrent neural networks

    James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural networks. arXiv preprint arXiv:1611.01576, 2016

    2016 arXiv

  57. [65]

    Retentive network: A successor to transformer for large language models, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023

    2023

  58. [66]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in neural information processing systems, pages 5998–6008, 2017. 14

    2017

  59. [67]

    Quantifying the carbon emissions of machine learning

    Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700, 2019

    1910 arXiv

  60. [68]

    Worst of both worlds: Biases compound in pre-trained vision-and-language models

    Tejas Srinivasan and Yonatan Bisk. Worst of both worlds: Biases compound in pre-trained vision-and-language models. arXiv preprint arXiv:2104.08666, 2021. 15 Checklist The checklist follows the references. Please read the checklist guidelines carefully for information on how t...

    2021 arXiv

  61. [69]

    For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] , in section 7 and in more detail in the supplementary material. (c) Did you d...

  62. [70]

    (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]

    If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]

  63. [71]

    for benchmarks)

    If you ran experiments (e.g. for benchmarks)... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We include the code, data and instructions needed in the supplement...

  64. [72]

    (a) If your work uses existing assets, did you cite the creators? [Yes]

    If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] . We heavily rely on Ego4D [3] annotations and cite their work accordingly. (b) Did you mention the license o...

  65. [73]

    If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (...

  66. [74]

    Recommended documentation frameworks include datasheets for datasets, dataset nutrition labels, data statements for NLP, and accountability frameworks

    Submission introducing new datasets must include the following in the supplementary materials: (a) Dataset documentation and intended uses. Recommended documentation frameworks include datasheets for datasets, dataset nutrition labels, data statements for NLP, and accountabili...

  67. [75]

    • Both data and metadata are hosted in the Github repository associated with the website github.com/sdqesdataset/sdqesdataset.github.io/

    To ensure accessibility, the supplementary materials fordatasets must include the following: (a) Links to access the dataset and its metadata. • Both data and metadata are hosted in the Github repository associated with the website github.com/sdqesdataset/sdqesdataset.github.i...

  68. [76]

    batch dimension

    For benchmarks, the supplementary materials must ensure that all results are easily repro- ducible. Where possible, use a reproducibility framework such as the ML reproducibility checklist, or otherwise guarantee that all results can be easily reproduced, i.e. all necessary da...

  69. [77]

    query":

    (i.e. d′ = d 2 ). Where we can, we set hyperparameters to be the same across all models trained. Other than the number of frames considered during training we also adjust the learning rate, reducing it when encountering training instabilities. Table 5 includes links to the log...

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.