Pith. sign in

REVIEW 3 major objections 5 minor 68 references

DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Processing only the half of video clips most relevant to a text query, as ranked by a cheap sidekick encoder, cuts compute by up to 47% and still improves temporal-grounding accuracy.

desk verdict A solid, well-ablated efficiency win for long-video grounding; the 47% savings claim needs an encoder-only qualifier and the saliency coverage worry is softer than it looks. read the letter →

arxiv 2505.16376 v1 pith:4KK25IW3 submitted 2025-05-22 cs.CV cs.AI

Zijia Lu , A S M Iftekhar , Gaurav Mittal , Tianjian Meng , Xiawei Wang , Cheng Zhao , Rohith Kukkala , Ehsan Elhamifar
show 1 more author
Mei Chen
This is my paper
classification cs.CVcs.AI
keywords temporalgroundinglongvideovideo-textretrievalsaliencyselectionefficientunderstandingdualencoderdelegate-and-conquerlocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DeCafNet claims that long-video temporal grounding does not need to run a large pre-trained encoder over every clip. It delegates dense feature extraction to a cheap sidekick encoder that also scores each clip's relevance to the text query, then runs the expensive expert encoder only on the top 30–50% of clips. The paper introduces DeCaf-Grounder to combine the two feature sets, which have different temporal resolutions, and reports that this scheme outperforms prior state-of-the-art methods on Ego4D-NLQ and Ego4D-Goalstep while cutting encoder computation by up to 47%.

What carries the argument

The central mechanism is the saliency map $S = F_D \cdot q_{\text{cls}} \in \mathbb{R}^T$, the inner product between the sidekick encoder's dense clip features and the text query's CLS token, which ranks all clips for selection of the top-c% to send to the expert encoder. The second mechanism is DeCaf-Grounder, which zero-pads the expert's salient features to align their temporal length with the dense features, concatenates both with the saliency scores, then applies query-aware temporal aggregation and multi-scale temporal refinement (dilated convolutions over a feature pyramid) so the grounding heads can reason jointly over the two resolutions.

What would settle it

Measure the cheap encoder's hit rate at top-50% selection on a long-video benchmark whose correct moments are spread evenly through the video or overlap several simultaneous events; if the hit rate falls below the 80.5% reported in Table 8, processing only the top half of clips would lose its accuracy edge over methods that encode every clip.

Watch

Extended reading notes

Core claim

On the Ego4D-NLQ benchmark, DeCafNet-50% reaches an average recall of 24.44 versus 23.14 for SnAG and 21.81 for RGNet, while on Ego4D-Goalstep it reaches 31.61 versus 30.49 for RGNet. Relative to the cost of processing every clip with the expert encoder, DeCafNet-50% cuts encoder TFLOPs by 47%, GPU memory by 44%, and inference time by 51%. The sidekick encoder uses convolution pooling and temporal interpolation to achieve a 31x TFLOP reduction over the expert encoder. Even with only 30% of clips sent to the expert encoder, DeCafNet roughly matches or beats prior full-encoder methods, and on MAD, where the grounding module is trained on pre-extracted features alone, DeCaf-Grounder raises the average recall to 16.47 versus 13.84 for SnAG.

Load-bearing premise

The cheap encoder's relevance scores rank the video clips well enough that sending only the top half of them to the full-size encoder rarely throws away the clip containing the correct moment, leaving the cheap encoder's lower-resolution features to handle the misses.

Editorial extensions

If this is right

  • Processing only the top 50% of salient clips with the expert encoder is enough to beat all prior methods on both Ego4D benchmarks while reducing encoder computation by 47%.
  • Even the more aggressive 30% selection ratio matches or improves on the prior best method, suggesting that a large fraction of clips can be cheaply coded without harming grounding.
  • DeCaf-Grounder used alone, with the same input features as SnAG, improves average recall by 1.9% on Ego4D-NLQ, showing that the grounding module contributes independent of the efficiency gains.
  • On MAD, which provides only pre-extracted features and no source video for training a sidekick encoder, DeCaf-Grounder still establishes a new state of the art, indicating the design transfers to fixed feature inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's margin rests on the sidekick's saliency recall: at 50% selection it includes only 80.5% of ground-truth clips, so the paper implicitly bets that the cheap dense features can carry the roughly one-in-five samples where the expert never sees the true clip; datasets with sparser or multi-moment videos could break that bet.
  • The same delegate-and-conquer pattern should transfer to other expensive per-item computations, such as long-context language modeling or high-resolution image analysis, wherever a cheap ranker can first identify the salient subset.
  • The 31x compute gap between sidekick and expert suggests the expert encoder is over-provisioned for most clips; a natural test is to distill the expert into the sidekick more aggressively and push the selection ratio below 30%, which the paper's own 80.5% recall figure suggests would be risky.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. DeCafNet proposes a dual-encoder architecture for long-video temporal grounding: a cheap sidekick encoder covers all clips (processing a subset and interpolating the rest) and generates a query-conditioned saliency map; a frozen expert encoder processes only the top-c% salient clips; and a DeCaf-Grounder fuses the dense and salient features via query-aware temporal aggregation and multi-scale temporal refinement. On Ego4D-NLQ, Ego4D-Goalstep, and MAD, the paper reports higher average recall than SnAG and RGNet while reducing encoder TFLOPs by up to 47% at the 50% selection ratio. The authors also evaluate DeCaf-Grounder on short-video grounding datasets and release code.

Significance. If the reported results hold, the paper demonstrates a practically useful efficiency-accuracy Pareto improvement: query-conditioned saliency from a cheap encoder can route expensive expert computation to relevant clips, and the grounding module can fuse heterogeneous-resolution features. Strengths include ablations that isolate the contributions of sidekick features, saliency scores, and DeCaf-Grounder components; a controlled comparison of DeCaf-Grounder against SnAG using identical input features (Table 10); robustness tests under degradation; and a public code release. The main caveat is that the central efficiency-accuracy claim rests on saliency-selection quality, which is currently characterized only by a per-clip recall statistic rather than by a query-level coverage measure.

major comments (3)
  1. [§4.4, Table 8] The saliency quality measure reported in Table 8 is a per-clip statistic: 'the percentage of ground truth clips present within the top 50% salient clips.' The paper's Pareto claim depends on the expert encoder not systematically missing the target moment, but the paper never reports the fraction of queries for which no ground-truth clip is selected. Since an average moment spans several clips, a clip-level recall of 80.5% is compatible with a high query-level coverage, but it could also hide a non-negligible fraction of queries whose target is never processed by the expert encoder; in that regime the prediction depends entirely on lower-resolution, partly interpolated sidekick features, which are not validated for this failure mode. Please report query-level coverage (percentage of queries with at least one ground-truth clip in the selected set) for both selection ratios, and analyze performance on queries where the expert encoder sees no true clip.
  2. [§3.3, Eqs. (2) and (3)] Equations (2) and (3) are written as ratios of positive-similarity exponentials over sums of positive and negative exponentials. As written, these are conditional probabilities that would be maximized, not losses to be minimized; minimizing them would push positive pairs apart, which contradicts the surrounding text. Presumably the authors intend a negative log-likelihood or InfoNCE-style objective, but the paper is internally inconsistent and not reproducible as printed. Please correct the definitions and state precisely how positive and negative pairs are sampled for both the text-side and video-side terms.
  3. [Tables 1, 3, 4, and ablations] No error bars, confidence intervals, or multiple-seed results are reported. The claimed state-of-the-art margins are small (for example, AVG 24.44 vs. 23.14 in Table 1 and 31.61 vs. 30.49 in Table 3), and several ablation differences are below one point. Without variance estimates it is not possible to determine whether these differences are statistically reliable. Please report means and standard deviations over at least three seeds for the main comparisons and for the key ablations in Tables 6, 7, and 9.
minor comments (5)
  1. [§4.3] The main-text references to 'Table 13' and 'Table 14' for the Ego4D-NLQ and Ego4D-Goalstep results should be to Table 1 and Table 3 of the main paper.
  2. [Table 2] The caption says Columns 1 and 2 show the amount of clips processed by each encoder, but the sidekick row labeled '100%' actually means 100% temporal coverage with only every other clip directly processed (τ=2) and the remaining features interpolated. Please clarify this in the caption.
  3. [Eq. (1)] The notation f'_2, ..., f'_{1+τ-1} = FFN([f'_1, f'_{1+τ}]) is not a well-formed assignment; please index the interpolated features explicitly, e.g., f'_k = FFN_k(f'_1, f'_{1+τ}) for k between sampled indices.
  4. [§4.2] The statement that the contrastive and distillation loss weights are '1 and 0.75' does not specify whether these multiply the summed losses or per-sample averages; please state the exact training objective and batch-level sampling procedure.
  5. [Supplementary Tables 13 and 14] The text refers to rows 'in blue', but the supplementary material is not color-coded; please use explicit row labels such as 'with NaQ pretraining' instead.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DeCafNet's efficiency and accuracy claims are supported by explicit cost arithmetic and held-out benchmark evaluations, not by self-referential definitions.

full rationale

DeCafNet is an empirical systems paper whose central claims—reduced computation and improved grounding accuracy—are validated against external benchmarks and explicit cost measurements, not derived from its own assumptions. The 47% TFLOP reduction in Table 2 is a direct arithmetic consequence of the reported per-encoder costs: sidekick encoder processes all clips (21.6 TFLOPs) plus expert encoder processes top-50% clips, versus prior methods processing all clips with the expert encoder (668.2 TFLOPs). This is transparent accounting, not circular reasoning. The accuracy claims are evaluated on held-out validation splits of Ego4D-NLQ, Ego4D-Goalstep, and MAD, with comparisons to prior methods under matched settings. The saliency-selection mechanism is not circular either: the sidekick encoder's saliency scores are trained with a contrastive loss using ground-truth clip-query pairs, and their quality is independently tested against random and uniform selection baselines (Table 7) and reported as clip-level recall (Table 8). The distillation loss from the frozen expert encoder is a training auxiliary, not a device that makes the evaluation metric equal to a training target. No load-bearing self-citation chain or imported uniqueness theorem appears; citations to the authors' prior works are contextual related-work references. Therefore, the derivation chain is self-contained with respect to the paper's stated evidence, and no circular step is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a set of empirical assumptions about video statistics (short moments, adjacent clip similarity) and about the reliability of the sidekick encoder's saliency signal. There are no ad hoc physical entities. The free parameters are architecture and training choices that control the reported efficiency-accuracy tradeoff.

free parameters (5)
  • Saliency selection ratio c = 30% or 50%
    Chosen by the authors and directly controls the efficiency-accuracy tradeoff; the abstract's "up to 47% reduction" refers to the 50% setting while the 30% setting gives a larger reduction but slightly weaker performance.
  • Temporal interpolation stride tau = 2
    Set to 2 after ablation (Table 8); processes every other clip and interpolates the rest, a key source of sidekick efficiency.
  • Convolution pooling location i = first block (i=1)
    Chosen empirically to balance sidekick recall and FLOPs; ablated in Table 8 rows 1-3.
  • Loss weights for saliency and distillation = 1.0 and 0.75
    Hand-set weights for L_saliency and L_distill; no sensitivity analysis is reported.
  • Number of multi-scale refinement scales L = 8
    Architectural choice for DeCaf-Grounder; also 8 dilation layers in the temporal convolution. No ablation on L is shown.
assumptions (5)
  • domain assumption Ground-truth moments constitute a small fraction of long videos (1.7% for Ego4D-NLQ, 2.2% for Ego4D-Goalstep), so top-c% clip selection can retain relevant content.
    Stated in Sec. 4.1 and used to justify delegating most clips to the cheap encoder. If moments were long or frequent, the saliency selection premise would weaken.
  • domain assumption Adjacent clips in a video contain similar content, so features of skipped clips can be interpolated from sampled neighbors (Eq. 1).
    This is the basis for temporal interpolation with tau=2 in Sec. 3.2. It may fail for fast cuts or brief moments.
  • domain assumption The sidekick encoder's features, after 4x spatial and temporal pooling and interpolation, are informative enough to both score saliency and support final grounding.
    The whole delegate-and-conquer scheme relies on this; Table 8 shows top-50% selection covers only 80.5% of ground-truth clips.
  • domain assumption Inner product similarity between clip features and the text CLS token is a valid saliency measure after contrastive training.
    Used in Sec. 3.4 (S = F_D dot q_cls). The validity is empirical and depends on the contrastive alignment learned by the saliency loss.
  • domain assumption The pretrained expert encoder features are a reliable teacher for distillation and a strong feature source for grounding.
    The distillation loss (Eq. 4) and the use of the frozen expert encoder from prior work assume the expert is high quality and that sidekick features can be aligned to it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos." pith.science (2026). https://pith.science/paper/4KK25IW3

@misc{pith2026250516376,
  author       = {Pith},
  title        = {Pith review of: DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4KK25IW3}},
  note         = {Machine review of arXiv:2505.16376}
}
read the original abstract

Long Video Temporal Grounding (LVTG) aims at identifying specific moments within lengthy videos based on user-provided text queries for effective content retrieval. The approach taken by existing methods of dividing video into clips and processing each clip via a full-scale expert encoder is challenging to scale due to prohibitive computational costs of processing a large number of clips in long videos. To address this issue, we introduce DeCafNet, an approach employing ``delegate-and-conquer'' strategy to achieve computation efficiency without sacrificing grounding performance. DeCafNet introduces a sidekick encoder that performs dense feature extraction over all video clips in a resource-efficient manner, while generating a saliency map to identify the most relevant clips for full processing by the expert encoder. To effectively leverage features from sidekick and expert encoders that exist at different temporal resolutions, we introduce DeCaf-Grounder, which unifies and refines them via query-aware temporal aggregation and multi-scale temporal refinement for accurate grounding. Experiments on two LTVG benchmark datasets demonstrate that DeCafNet reduces computation by up to 47\% while still outperforming existing methods, establishing a new state-of-the-art for LTVG in terms of both efficiency and performance. Our code is available at https://github.com/ZijiaLewisLu/CVPR2025-DeCafNet.

Figures

Figures reproduced from arXiv: 2505.16376 by the authors.

Figure 1
Figure 1. Model inference time and grounding performance on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. DeCafNet Overview. The sidekick encoder efficiently extracts features from input video clips, which, combined with text [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. DeCafNet’s qualitative results, where Ours wo DCG and Ours w DCG indicate predictions without and with DeCaf-Grounder, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Details of multi-scale temporal refinement. The multi-scale features produced by the temporal transformer are transformed into [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 58 canonical work pages

  1. [1]

    Localizing Moments in Long Video Via Multimodal Guidance

    Wayner Barrios, Mattia Soldan, Fabian Caba Heilbron, Al- berto M. Ceballos-Arroyo, and Bernard Ghanem. Localiz- ing moments in long video via multimodal guidance.ArXiv, abs/2302.13372, 2023. 6

    work page Pith review arXiv 2023

  2. [2]

    Is space-time attention all you need for video understanding? InProceedings of the International Conference on Machine Learning (ICML), 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? InProceedings of the International Conference on Machine Learning (ICML), 2021. 3

    work page 2021

  3. [3]

    Acceler- ating large language model decoding with speculative sam- pling.arXiv preprint arXiv:2302.01318, 2023

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Baptiste Lespiau, Laurent Sifre, and John Jumper. Acceler- ating large language model decoding with speculative sam- pling.arXiv preprint arXiv:2302.01318, 2023. 2

    arXiv 2023

  4. [4]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InProceedings of the 37th Interna- tional Conference on Machine Learning, pages 1597–1607,

  5. [5]

    Tallformer: Temporal ac- tion localization with a long-memory transformer

    Feng Cheng and Gedas Bertasius. Tallformer: Temporal ac- tion localization with a long-memory transformer. InEu- ropean Conference on Computer Vision, pages 503–521. Springer, 2022. 2

    work page 2022

  6. [6]

    Every mistake counts in assembly.arXiv preprint arXiv:2307.16453, 2023

    Guodong Ding, Fadime Sener, Shugao Ma, and Angela Yao. Every mistake counts in assembly.arXiv preprint arXiv:2307.16453, 2023. 2

    arXiv 2023

  7. [7]

    Coherent temporal synthesis for incremental action segmentation

    Guodong Ding, Hans Golong, and Angela Yao. Coherent temporal synthesis for incremental action segmentation. In 2024 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2024

    work page 2024

  8. [8]

    Donahue and E

    G. Donahue and E. Elhamifar. Learning to predict activity progress by self-supervised video alignment.IEEE Confer- ence on Computer Vision and Pattern Recognition, 2024

    work page 2024

Show all 68 references
  1. [9]

    Ms-tcn: Multi-stage tem- poral convolutional network for action segmentation

    Yazan Abu Farha and Jurgen Gall. Ms-tcn: Multi-stage tem- poral convolutional network for action segmentation. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3575–3584, 2019. 2, 5

    2019

  2. [10]

    Tall: Temporal activity localization via language query,

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Neva- tia. Tall: Temporal activity localization via language query,

  3. [11]

    Mac: Mining activity concepts for language-based temporal local- ization, 2018

    Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia. Mac: Mining activity concepts for language-based temporal local- ization, 2018. 2

    2018

  4. [12]

    Diverse sequential subset selection for supervised video summarization

    Boqing Gong, Wei-Lun Chao, Kristen Grauman, and Fei Sha. Diverse sequential subset selection for supervised video summarization. InAdvances in Neural Information Process- ing Systems, pages 2069–2077, 2014. 1

    2014

  5. [13]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. pages 18995–19012, 2022. 1, 2, 5

    2022

  6. [14]

    Rehg, and Hans Peter Graf

    Meera Hahn, Asim Kadav, James M. Rehg, and Hans Peter Graf. Tripping through time: Efficient localization of activi- ties in videos, 2020. 2

    2020

  7. [15]

    Rgnet: A unified clip retrieval and grounding network for long videos

    Tanveer Hannan, Md Mohaiminul Islam, and Thomas Seidl. Rgnet: A unified clip retrieval and grounding network for long videos. InEuropean Conference on Computer Vision,

  8. [16]

    Localizing mo- ments in video with natural language, 2017

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing mo- ments in video with natural language, 2017. 2

    2017

  9. [17]

    Cone: An efficient coarse-to-fine alignment frame- work for long video temporal grounding.arXiv preprint arXiv:2209.10918, 2022

    Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, and Nan Duan. Cone: An efficient coarse-to-fine alignment frame- work for long video temporal grounding.arXiv preprint arXiv:2209.10918, 2022. 1, 2, 5, 6

    2022 arXiv

  10. [18]

    Content-based recommendation engine for video streaming platform.arXiv preprint arXiv:2308.08406, 2023

    Puskal Khadka and Prabhav Lamichhane. Content-based recommendation engine for video streaming platform.arXiv preprint arXiv:2308.08406, 2023. 1

    2023

  11. [19]

    Lost in time: Temporal analytics for long-term video surveillance.arXiv preprint arXiv:1712.07322, 2017

    Huai-Qian Khor and John See. Lost in time: Temporal analytics for long-term video surveillance.arXiv preprint arXiv:1712.07322, 2017. 1

    2017 arXiv

  12. [20]

    S. Lee, Z. Lu, Z. Zhang, M. Hoai, and E. Elhamifar. Error detection in egocentric procedural task videos.IEEE Con- ference on Computer Vision and Pattern Recognition, 2024. 2

    2024

  13. [21]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34: 11846–11858, 2021. 2, 6

    2021

  14. [22]

    Progressive video summarization via multimodal self- supervised learning

    Haopeng Li, Qiuhong Ke, Mingming Gong, and Tom Drum- mond. Progressive video summarization via multimodal self- supervised learning. InProceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision (WACV), pages 5584–5593, 2023. 1

    2023

  15. [23]

    Vigt: proposal-free video grounding with a learnable token in the transformer

    Kun Li, Dan Guo, and Meng Wang. Vigt: proposal-free video grounding with a learnable token in the transformer. Science China Information Sciences, 66(10), 2023. 2

    2023

  16. [24]

    Egocentric video-language pretraining.arXiv preprint arXiv:2206.01670, 2022

    Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Sol- dan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining.arXiv preprint arXiv:2206.01670, 2022. 5, 2

    2022 arXiv

  17. [25]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2794–28...

    2023

  18. [26]

    Solving masked jigsaw puzzles with diffusion vision transformers

    Jinyang Liu, Wondmgezahu Teshome, Sandesh Ghimire, Mario Sznaier, and Octavia Camps. Solving masked jigsaw puzzles with diffusion vision transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23009–23018, 2024. 2

    2024

  19. [27]

    Lu and E

    Z. Lu and E. Elhamifar. Weakly-supervised action segmenta- tion and alignment via transcript-aware union-of-subspaces learning.International Conference on Computer Vision, 2021

    2021

  20. [28]

    Lu and E

    Z. Lu and E. Elhamifar. Set-supervised action learning in procedural task videos via pairwise order consistency.IEEE Conference on Computer Vision and Pattern Recognition,

  21. [29]

    Lu and E

    Z. Lu and E. Elhamifar. Fact: Frame-action cross-attention temporal modeling for efficient action segmentation.IEEE Conference on Computer Vision and Pattern Recognition,

  22. [30]

    Self-supervised multi-object tracking with path consistency

    Zijia Lu, Bing Shuai, Yanbei Chen, Zhenlin Xu, and Davide Modolo. Self-supervised multi-object tracking with path consistency. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19016–19026, 2024. 2

    2024

  23. [31]

    Snag: Scalable and accurate video grounding

    Fangzhou Mu, Sicheng Mo, and Yin Li. Snag: Scalable and accurate video grounding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18930–18940, 2024. 2, 3, 5, 6, 7

    2024

  24. [32]

    A content-driven micro-video recommendation dataset at scale

    Yongxin Ni, Yu Cheng, Xiangyan Liu, Junchen Fu, Youhua Li, Xiangnan He, Yongfeng Zhang, and Fajie Yuan. A content-driven micro-video recommendation dataset at scale. arXiv preprint arXiv:2309.15379, 2023. 1

    2023 arXiv

  25. [33]

    Scanning only once: An end-to-end framework for fast temporal grounding in long videos

    Yulin Pan, Xiangteng He, Biao Gong, Yiliang Lv, Yujun Shen, Yuxin Peng, and Deli Zhao. Scanning only once: An end-to-end framework for fast temporal grounding in long videos. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 13767–13777, 2023. ...

    2023

  26. [34]

    Category-specific video summarization

    Danila Potapov, Matthijs Douze, Zaid Harchaoui, and Cordelia Schmid. Category-specific video summarization. In European Conference on Computer Vision, pages 540–555,

  27. [35]

    Ramakrishnan, Ziad Al-Halah, and Kristen Grauman

    Santhosh K. Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Naq: Leveraging narrations as queries to su- pervise episodic memory. InComputer Vision and Pat- tern Recognition (CVPR), 2023 IEEE Conference on. IEEE,

    2023

  28. [36]

    Ground- ing action descriptions in videos.Transactions of the Asso- ciation for Computational Linguistics, 1:25–36, 2013

    Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Ground- ing action descriptions in videos.Transactions of the Asso- ciation for Computational Linguistics, 1:25–36, 2013. 2

    2013

  29. [37]

    Ground- ing action descriptions in videos.Transactions of the Asso- ciation for Computational Linguistics, 2013

    Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Ground- ing action descriptions in videos.Transactions of the Asso- ciation for Computational Linguistics, 2013. 7

    2013

  30. [38]

    Hat: History-augmented anchor transformer for on- line temporal action localization

    Sakib Reza, Yuexi Zhang, Mohsen Moghaddam, and Octavia Camps. Hat: History-augmented anchor transformer for on- line temporal action localization. InEuropean Conference on Computer Vision, pages XXX–XXX. Springer, 2024. 2

    2024

  31. [39]

    Shen and E

    Y . Shen and E. Elhamifar. Progress-aware online action seg- mentation for egocentric procedural task videos.IEEE Con- ference on Computer Vision and Pattern Recognition, 2024. 2

    2024

  32. [40]

    InHollywood in Homes: Crowdsourcing Data Collection for Activity Understanding,

    Gunnar Sigurdsson, G ¨ul Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. InHollywood in Homes: Crowdsourcing Data Collection for Activity Understanding,

  33. [41]

    Sigurdsson, G ¨ul Varol, X

    Gunnar A. Sigurdsson, G ¨ul Varol, X. Wang, Ali Farhadi, Ivan Laptev, and Abhinav Kumar Gupta. Hollywood in homes: Crowdsourcing data collection for activity under- standing. InEuropean Conference on Computer Vision,

  34. [42]

    Mad: A scalable dataset for language grounding in videos from movie audio descriptions

    Mattia Soldan, Alejandro Pardo, Juan Le ´on Alc´azar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem. Mad: A scalable dataset for language grounding in videos from movie audio descriptions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2022

  35. [43]

    Multimodal sparse transformer network for audio-visual speech recognition

    Qiya Song, Bin Sun, and Shutao Li. Multimodal sparse transformer network for audio-visual speech recognition. IEEE Transactions on Neural Networks and Learning Sys- tems, 34(12):10028–10038, 2022. 2

    2022

  36. [44]

    Ego4d goal-step: To- ward hierarchical understanding of procedural activities

    Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: To- ward hierarchical understanding of procedural activities. In Advances in Neural Information Processing Systems, 2023. 5, 6, 1

    2023

  37. [45]

    Two-stage active learning for efficient temporal action segmentation

    Yuhao Su and Ehsan Elhamifar. Two-stage active learning for efficient temporal action segmentation. InEuropean Con- ference on Computer Vision, pages 161–183. Springer, 2024. 2

    2024

  38. [46]

    Structured multi-level interaction network for video moment localization via language query

    Hao Wang, Zheng-Jun Zha, Liang Li, Dong Liu, and Jiebo Luo. Structured multi-level interaction network for video moment localization via language query. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7022–7031, 2021. 7

    2021

  39. [47]

    Tempo- rally grounding language queries in videos by contextual boundary-aware prediction, 2019

    Jingwen Wang, Lin Ma, and Wenhao Jiang. Tempo- rally grounding language queries in videos by contextual boundary-aware prediction, 2019. 2

    2019

  40. [48]

    Language- driven temporal activity localization: A semantic matching reinforcement learning model

    Weining Wang, Yan Huang, and Liang Wang. Language- driven temporal activity localization: A semantic matching reinforcement learning model. In2019 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 334–343, 2019. 2

    2019

  41. [49]

    Proposal relation network for temporal ac- tion detection, 2021

    Xiang Wang, Zhiwu Qing, Ziyuan Huang, Yutong Feng, Shi- wei Zhang, Jianwen Jiang, Mingqian Tang, Changxin Gao, and Nong Sang. Proposal relation network for temporal ac- tion detection, 2021. 2

    2021

  42. [50]

    Video- groundingdino: Towards open-vocabulary spatio-temporal video grounding, 2024

    Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Video- groundingdino: Towards open-vocabulary spatio-temporal video grounding, 2024. 2

    2024

  43. [51]

    Explore-and-match: Bridging proposal-based and proposal-free with transformer for sentence grounding in videos, 2022

    Sangmin Woo, Jinyoung Park, Inyong Koo, Sumin Lee, Minki Jeong, and Changick Kim. Explore-and-match: Bridging proposal-based and proposal-free with transformer for sentence grounding in videos, 2022. 2

    2022

  44. [52]

    Multi-modal circulant fusion for video-to-language and backward

    Aming Wu and Yahong Han. Multi-modal circulant fusion for video-to-language and backward. InProceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 1029–1035. International Joint Conferences on Artificial Intelligence Organi...

    2018

  45. [53]

    Long-term feature banks for detailed video understanding, 2019

    Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaim- ing He, Philipp Kr ¨ahenb¨uhl, and Ross Girshick. Long-term feature banks for detailed video understanding, 2019. 2

    2019

  46. [54]

    Efficient and effec- tive weakly-supervised action segmentation via action- transition-aware boundary alignment

    Angchi Xu and Wei-Shi Zheng. Efficient and effec- tive weakly-supervised action segmentation via action- transition-aware boundary alignment. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18253–18262, 2024. 2

    2024

  47. [55]

    Hauptmann

    Shoou-I Yu, Yi Yang, Xuanchong Li, and Alexander G. Hauptmann. Long-term identity-aware multi-person track- ing for surveillance video summarization.arXiv preprint arXiv:1604.07468, 2016. 1

    2016 arXiv

  48. [56]

    Dense regression network for video grounding, 2020

    Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan. Dense regression network for video grounding, 2020. 2

    2020

  49. [57]

    Actionformer: Localizing moments of actions with transformers

    Chenlin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. InPro- ceedings of the European Conference on Computer Vision (ECCV), pages 492–510, 2022. 5

    2022

  50. [58]

    Helping hands: An object-aware ego-centric video recogni- tion model

    Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Helping hands: An object-aware ego-centric video recogni- tion model. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 13901–13912, 2023. 6

    2023

  51. [59]

    Span-based localizing network for natural language video lo- calization.arXiv preprint arXiv:2004.13931, 2020

    Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video lo- calization.arXiv preprint arXiv:2004.13931, 2020. 1, 2, 3, 6

    2004 arXiv

  52. [60]

    Multi-stage aggregated transformer network for temporal language localization in videos

    Mingxing Zhang, Yang Yang, Xinghan Chen, Yanli Ji, Xing Xu, Jingjing Li, and Heng Tao Shen. Multi-stage aggregated transformer network for temporal language localization in videos. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 7

    2021

  53. [61]

    Learning 2d temporal adjacent networks for moment local- ization with natural language

    Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment local- ization with natural language. InProceedings of the AAAI Conference on Artificial Intelligence, pages 12870–12877,

  54. [62]

    OnlineTAS: An online baseline for temporal action segmentation

    Qing Zhong, Guodong Ding, and Angela Yao. OnlineTAS: An online baseline for temporal action segmentation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 2

    2024

  55. [63]

    Enriching local and global contexts for temporal action localization, 2021

    Zixin Zhu, Wei Tang, Le Wang, Nanning Zheng, and Gang Hua. Enriching local and global contexts for temporal action localization, 2021. 2 DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos Supplementary Material We present DeCafNet, an efficient algo...

    2021

  56. [64]

    DeCaf-Grounder consists of the following key components: query-aware temporal aggregation, multi- scale temporal refinement, and classifier & regressor

    Additional Architectural Details To enable temporal grounding using features extracted by both the sidekick and expert encoders, we introduce DeCaf- Grounder. DeCaf-Grounder consists of the following key components: query-aware temporal aggregation, multi- scale temporal refin...

  57. [65]

    In Table 12 of this supplementary material, we also show the computation on Ego4D-Goalstep dataset

    Computation Efficiency on Ego4D-Goalstep In Table 2 of the main paper, we have reported compu- tation efficiency on Ego4D-NLQ dataset. In Table 12 of this supplementary material, we also show the computation on Ego4D-Goalstep dataset. Row 2 shows the feature ex- traction cost ...

  58. [66]

    Their settings are consis- Figure 4

    Additional Experimental Results Table 13, 14 show complete model results on Ego4D-NLQ and Ego4D-Goalstep datasets. Their settings are consis- Figure 4. Details of multi-scale temporal refinement. The multi-scale features produced by the temporal transformer are transformed int...

  59. [67]

    For tem- poral convolution [29] in multi-scale temporal refinement, we use 8 layers, where the dilation rate of thei-th convolu- tion layer equals to2 i

    Implementation Details Our sidekick encoder has 12 spatio-temporal blocks and we initialize its weight from [24] to speed up training. For tem- poral convolution [29] in multi-scale temporal refinement, we use 8 layers, where the dilation rate of thei-th convolu- tion layer eq...

  60. [68]

    Where was object X before I used it?

    Limitations DeCafNet has established new SOTA for LVTG with greatly reduced computation. However, the overall re- call values are relatively low, especially for R1@0.3 and R1@0.5. We found this is partly caused by ambiguity in text queries in the dataset. For example, for a te...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.