Pith. sign in

REVIEW 4 major objections 5 minor 91 references

Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read OpenMixer, a query-based detector built on a frozen video-language model, detects and names human actions never seen in training, beating all baselines on J-HMDB and UCF101-24.

desk verdict A solid, well-ablated entry to open-vocabulary action detection; the VLM-attention location prior is the real idea, though the evaluation is narrow and the concurrent comparison is missing. read the letter →

arxiv 2411.10922 v1 pith:CKD32DP6 submitted 2024-11-17 cs.CV

classification cs.CV
keywords open-vocabularyactiondetectionvideovision-languagemodelstransformerquery-baseddetectorzero-shotrecognitionactorlocalizationprompt-basedclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-vocabulary action detection means finding and naming actions in video that a model never saw in training. The paper argues this is achievable without fine-tuning a large vision-language model: a DETR-style detector, OpenMixer, harvests two resources from a frozen video VLM (the localizability of its patch-text attention, which shows where actors are, and the semantics of its video-level feature, which carries what the action is). The spatial block samples initial box centers from the VLM attention map, the temporal block conditions query decoding on the video feature, and a dynamic fusion module blends pre-trained semantics into learnable queries matched against GPT-generated action descriptions. On J-HMDB the model reaches 86.34 mean video mAP, with 82.33 on novel actions; on UCF101-24 it reaches 47.71 mean, with 34.23 on novel actions, outperforming all baselines. If this holds, action detection in an open world no longer needs region-level pre-training or annotated supervision for every possible action.

What carries the argument

The load-bearing mechanism is the reversed patch-text attention map $\hat{S} = 1 - V \otimes f_t$: the video VLM's patch features $V$ are correlated with the text feature $f_t$ of the single class whose embedding most closely matches the global video feature $f_v$, and the map is reversed because CLIP-style attention pooling produces high patch-text similarity on background regions. Top-$\hat{S}$ positions supply the initial box centers for the spatial queries, turning "where is the actor?" into a cheap read-out from a frozen model instead of a learned proposal stage. Two further mechanisms carry the recognition side: the temporal block broadcasts $f_v$ into the query-video mixing (an AdaMixer-style design) so decoding is conditioned on test-time video content rather than class text, and the DFA module learns a query-specific scalar $\lambda$ to interpolate between the frozen video feature and the learnable query before cosine-similarity classification against GPT-generated action descriptions.

What would settle it

Measure the localization quality of the prior directly: take a held-out video set with actor bounding-box annotations, pick novel (unseen) action classes, compute the reversed patch-text attention of the frozen VLM for each video, and count how often the top-100 sampled positions fall inside a ground-truth actor box. If the hit rate is at or below chance for clips with small actors or subtle motion, the location-prior assumption fails where the paper's novel-class gains are claimed, and the reported localization cannot be coming from seeds derived from that map.

Watch

Extended reading notes

Core claim

The central claim is that a frozen video VLM contains two untapped resources (localizability and semantics) that together close the representation gap between video-level pre-training and region-level action detection. Concretely, OpenMixer uses the reversed patch-text correlation $\hat{S} = 1 - V \otimes f_t$, where $f_t$ is the text feature of the class most similar to the global video feature $f_v$, as a prior distribution over person locations, and samples the top-$N$ positions as initial box centers for the spatial OpenMixer Block (S-OMB). The temporal block (T-OMB) broadcasts the video-level feature into the query-video mixing; the dynamically fused alignment (DFA) learns a per-query weight $\lambda$ to fuse the frozen VLM's video feature with learnable query features, then classifies by cosine similarity against GPT-generated action descriptions. With localization decoupled from recognition (the spatial queries never touch the classifier), the model localizes class-agnostically and recognizes both base and novel actions, reporting 86.34 mean video mAP on J-HMDB and 47.71 on UCF101-24 in the end-to-end setting, ahead of all baselines.

Load-bearing premise

The load-bearing premise is that the frozen video-language model's patch-text attention, computed with the single action description that best matches the whole video, marks where the actors actually are for actions the model never trained on; if that attention points at the wrong locations, the box proposals start wrong and the class-agnostic localization cannot recover.

Editorial extensions

If this is right

  • Open-vocabulary action detection can be built on a frozen video VLM with a light DETR head: no region-level pre-training, no VLM fine-tuning, and no label supervision for novel actions.
  • Localization and recognition should be decoupled: feeding class text or spatial queries into the classifier hurts both base and novel performance, so class-agnostic localization is the right design for open vocabularies.
  • Descriptive, LLM-generated action prompts improve not just recognition but also temporal localization (temporal mAP rises from 83.57 to 91.62 on J-HMDB), so text quality is part of the detection pipeline.
  • The choice of VLM matters more than the head: swapping the video-pretrained CLIP-ViP for image CLIP drops mean mAP from 86.34 to 71.60, so motion-aware pre-training is a precondition for the approach.
  • Gains persist under generalized zero-shot testing with the full vocabulary given (74.28 versus 36.26 for the STMixer baseline on J-HMDB), not just when base and novel classes are scored separately.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A corollary the authors leave implicit: because random location priors collapse mean mAP from 86.34 to 54.15 while ground-truth priors push it to 91.19, the method's ceiling is set by how faithfully the VLM attention marks actors (improving the prior via multi-frame aggregation, ensembling over prompt descriptions, or attention across ViT layers may pay off more than architectural changes).
  • The recipe of seeding proposals from VLM patch-text attention, conditioning decoding on the global video feature, and fusing pre-trained semantics with a per-query weight should transfer to other supervision-starved region-level video tasks, such as open-vocabulary video grounding or referring-expression localization.
  • A testable extension: building the location prior from an ensemble of prompt descriptions rather than the single best-matching class text could improve novel-class localization for small or partially occluded actors, where one class match is most fragile.
  • The paper states that the large-scale AVA dataset is excluded; all evidence rests on two small benchmarks, so whether the gains survive on dense, multi-person, long-duration video is an open question the authors flag for future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces OpenMixer, a query-based DETR-style architecture for open-vocabulary action detection (OVAD) built on a frozen video vision-language model (CLIP-ViP). The method exploits the VLM's semantics through a temporal OpenMixer block (T-OMB) and a dynamically fused alignment (DFA) module, and its localizability through a spatial OpenMixer block (S-OMB) that initializes person-box centers from a reversed patch-text attention map. Experiments on J-HMDB and UCF101-24 under three protocols (zero-shot recognition with zero-shot localization, zero-shot recognition with trainable localization, and end-to-end training) report large improvements over an STMixer baseline using the same frozen VLM, together with component ablations and additional split-level results in the appendix.

Significance. If the reported results are robust, this is a valuable contribution to open-vocabulary action detection. The paper demonstrates that a frozen video VLM can supply both semantic and spatial priors to a learnable DETR-style head, and it contributes new benchmark protocols and public splits. The manuscript has notable strengths: the authors release code, models, and splits; they report five random splits for both 50%-50% and 75%-25% seen/unseen partitions; they include a generalized zero-shot evaluation in the appendix; and their ablations (Tables 3-5, 9) are carefully designed with upper- and lower-bound location priors. The central claims, however, depend on the assumption that the argmax-class patch-text attention is a spatially faithful prior for novel actions, and on a single-split comparison in the main table; both require additional evidence before the state-of-the-art claim is fully established.

major comments (4)
  1. [Sec. 3.2, Table 9] The claim that the argmax-class patch-text attention yields a faithful spatial prior for novel actions is central to the method, but the evidence in Table 9 does not isolate the failure mode. The gap between the ground-truth-prior upper bound (91.19) and the attention prior (86.34) is about 5 mAP points, and Table 9 reports no prior recall, no base/novel stratification, and no UCF101-24 results. I request an additional analysis: (i) report the recall of the top-N sampled centers against ground-truth boxes separately for base and novel classes; (ii) replace the global argmax class with the ground-truth class when constructing S to quantify how much of the gap is due to class-selection error versus patch-text spatial noise; and (iii) repeat the noise-level comparison on UCF101-24, including a multi-person subset. This is load-bearing because Table 3 shows that removing S-OMB reduces novel-class mAP from 82.33 to 77.64.
  2. [Sec. 4.1, Table 1, Appendix D] The main SOTA comparison in Table 1 is based on a single random class split and on an individual base/novel evaluation protocol, while Appendix Tables 11-14 show large split-to-split variation (e.g., novel mAP on UCF101-24 ranges from 28.55 to 61.96 under the 75%-25% split). No baseline is evaluated on the same alternative splits, so the reader cannot tell whether the reported improvements over STMixer are within the split noise. Please report the baselines on the same five 50%-50% splits (or at least provide standard deviations or error bars for Table 1), and consider making the generalized zero-shot protocol of Table 10 the primary evaluation in the main text, with individual testing as an auxiliary result.
  3. [Sec. 2, Appendix F] The concurrent work [78] addresses the same OVAD task, and the paper does not provide an empirical comparison. The reasons given in Appendix F (no released code, different splits and metrics) are understandable, but the absence of any quantitative positioning relative to the only other same-task method weakens the 'performs the best over baselines' claim. I ask the authors to add a direct comparison under a shared protocol if the data of [78] become available, or at least to include a table that translates their results to the J-HMDB/UCF101-24 protocol used here, and to state clearly in the abstract and introduction that the comparison excludes [78].
  4. [Table 5] The fixed-equal-fusion baseline (lambda=0.5) yields 51.48 mean mAP, which is far below both lambda=0 (68.84) and lambda=1 (74.06). This non-monotonic behavior is not explained by the paper's narrative about adaptivity, and it raises the possibility of an optimization or scaling artifact (e.g., feature magnitudes, initialization, or loss weighting). Please provide an explanation, report the training curves or convergence, and verify the result across random seeds. This is important because the DFA module's dynamic fusion is a core contribution.
minor comments (5)
  1. [Sec. 3.2] The operator symbol is used for both the inner product between fv and ft and between V and ft, which makes the equations ambiguous; please use a dot product or explicitly define the tensor contraction.
  2. [Sec. 4.1] The phrase 'more than 10% performance gain' should be 'more than 10 mAP points' for precision.
  3. [Introduction] There is a typo in 'the sate-of-the-art performance'; it should be 'state-of-the-art'.
  4. [Table 1 caption] The caption does not state how many random splits were used or which split is reported; please reference Appendix D explicitly in the caption.
  5. [Limitations and Future Work] The phrase 'we emphasize on the adaptation' is ungrammatical; consider 'we emphasize adaptation'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the open-vocabulary claim is an empirical result with train/novel class separation, and no derivation reduces to its own inputs.

full rationale

The paper is an empirical systems paper, not a formal derivation, and its central claim is evaluated on held-out novel action classes. The recognition path uses cosine similarity between learned query features and text features of base and novel classes; text features come from a frozen VLM and GPT-generated prompts and are not fitted to the training videos, so novel-class prediction is not forced by construction. The location-prior mechanism in Sec. 3.2 samples initial box centers from the frozen VLM's patch-text attention and then learns box offsets; this is architectural reuse of the same frozen backbone, not a parameter fitted to the test labels or a quantity definitionally equal to the output. The paper's own Table 9 probes the prior's contribution by comparing against a ground-truth upper bound and a random lower bound, which is an empirical sensitivity check rather than a circular reduction. The only self-referential element is that the same frozen VLM supplies both the attention prior and the recognition features, but this does not make any prediction equivalent to its inputs. The skeptical concern that the argmax-class patch-text attention may be unfaithful for novel actions is a legitimate limitation, and the paper itself notes the restricted scope in 'Limitations and Future Work' (small datasets, AVA omitted) and only tests prior noise on J-HMDB in Appendix Table 9; however, under-testing of an assumption is a correctness risk, not circularity. No load-bearing self-citation chain or imported uniqueness theorem exists; the reversed-attention choice from [32] is an externally cited empirical heuristic with ablative support. Accordingly, no circular step meets the evidentiary bar of Eq. X = Eq. Y by construction or fitted parameter renamed as prediction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the frozen VLM's semantics and localizability, on GPT-generated prompts, and on several hand-tuned or learned hyperparameters. The main free parameters are the per-query fusion weight lambda and the architecture sizes N and M; the main assumptions concern the VLM attention map's faithfulness and the transferability of fv to unseen actions.

free parameters (5)
  • dynamic fusion weight lambda (per-query) = learned (no closed form)
    A learnable scalar per query in DFA (Sec. 3.4) controls the blend of the frozen VLM's video feature fv with the temporal query feature; it is trained on base classes and applied to novel classes at test.
  • loss weights w1 and w2 = w1=2.0, w2=48.0
    Hand-chosen weights balancing the DETR set loss and action cross-entropy loss (Sec. 3.5); no sensitivity analysis is reported.
  • number of queries N and OMB stages M = N=100, M=3
    Selected by the hyperparameter sweep in Fig. 3a-b; the reported mAP depends on these choices.
  • person score threshold = 0.6
    Inference-time threshold used to keep boxes (Sec. 4 Implementation details); affects precision-recall trade-off.
  • GPT prompts per class = 16 for UCF101-24; 1 for J-HMDB
    Number of GPT-4-generated descriptive prompts per class, mean-pooled into text features; Appendix Tab. 7 shows prompts change temporal localization mAP.
assumptions (5)
  • domain assumption The frozen VLM's patch-text attention (reversed similarity) provides a faithful prior for where actors are in both base and novel actions.
    Invoked in Sec. 3.2 to sample initial box centers; if the attention map is unfaithful for a novel action, the spatial queries start from wrong locations and localization fails.
  • domain assumption The video-level feature fv of a frozen VLM transfers to unseen actions without adapting the VLM.
    Used as the semantic condition in T-OMB (Eq. 1) and in DFA fusion (Sec. 3.4); the whole open-vocabulary capability hinges on fv remaining discriminative for novel classes.
  • domain assumption GPT-4-generated text prompts are more transferable for open-vocabulary recognition than handcrafted templates.
    All recognition text features are mean-pooled GPT outputs (Appendix A); Tab. 7 shows GPT prompts improve temporal mAP, but the method depends on this being true for the action vocabulary at hand.
  • standard math DETR-style bipartite matching and set losses are appropriate for class-agnostic person localization.
    Imported from DETR/STMixer (Sec. 3.5) to supervise box regression independently of action class, a known technique.
  • ad hoc to paper The 50%-50% (and 75%-25%) random class splits of J-HMDB and UCF101-24 define a representative open-vocabulary evaluation.
    These splits are newly created for the OVAD benchmark (Sec. 4); results vary across splits (Appendix D, Tabs. 11-14), so the reported single-split numbers are split-dependent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection." pith.science (2026). https://pith.science/paper/CKD32DP6

@misc{pith2026241110922,
  author       = {Pith},
  title        = {Pith review of: Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKD32DP6}},
  note         = {Machine review of arXiv:2411.10922}
}
read the original abstract

Action detection aims to detect (recognize and localize) human actions spatially and temporally in videos. Existing approaches focus on the closed-set setting where an action detector is trained and tested on videos from a fixed set of action categories. However, this constrained setting is not viable in an open world where test videos inevitably come beyond the trained action categories. In this paper, we address the practical yet challenging Open-Vocabulary Action Detection (OVAD) problem. It aims to detect any action in test videos while training a model on a fixed set of action categories. To achieve such an open-vocabulary capability, we propose a novel method OpenMixer that exploits the inherent semantics and localizability of large vision-language models (VLM) within the family of query-based detection transformers (DETR). Specifically, the OpenMixer is developed by spatial and temporal OpenMixer blocks (S-OMB and T-OMB), and a dynamically fused alignment (DFA) module. The three components collectively enjoy the merits of strong generalization from pre-trained VLMs and end-to-end learning from DETR design. Moreover, we established OVAD benchmarks under various settings, and the experimental results show that the OpenMixer performs the best over baselines for detecting seen and unseen actions. We release the codes, models, and dataset splits at https://github.com/Cogito2012/OpenMixer.

Figures

Figures reproduced from arXiv: 2411.10922 by the authors.

Figure 1
Figure 1. Framework (left) and the OpenMixer Block (right). Given a video and an open vocabulary of actions, we use prompted classes and a pre-trained video VLM to obtain all kinds of VLM features. With a stack of cascaded OpenMixer blocks and spatial-temporal queries, the model predicts the action scores, person boxes, and their associated person scores for the OVAD task. works in object detection [25, 27, 35, 39, 79, 86, 87… view at source ↗
Figure 2
Figure 2. Spatial and Temporal OMB, and DFA. In Figs. 2a and 2b, the Q-Q and Q-V mixing modules aim to mix information among queries and across query-visual features, respectively. S-OMB is in Sec. 3.2 where the dashed arrow is only used at the 1st stage. T-OMB is in Sec. 3.3 and DFA is in Sec. 3.4. son boxes from the (m − 1)-th OMB, and predict person scores and action scores. These three modules are devel￾oped for the OVAD … view at source ↗
Figure 3
Figure 3. Hyperparameters. We show the video mAP with re￾spect to different numbers of learnable queries and OMB stages. dataset such as the long duration (∼ 10× longer), heavy background bias, and multi-person scenarios. Zero-shot action detection. We note the iCLIP [20] de￾fines the zero-shot action detection (ZSAD) task which is different from our OVAD task. The ZSAD only cares about the samples from novel classes while OV… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Unseen Action Detection. We visualize our OpenMixer detections (in blue) and ground truth (in yellow) on two representative videos from novel classes. The numbers after class names are confidence scores. More visualizations are in Appendix E. without S-OMB which means …
Figure 5
Figure 5. Figure 5: Generated prompts for J-HMDB action categories. For each category, we generate one prompt sentence. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Generated prompts for UCF101-24 action categories. For each category, we generate 16 prompt sentences. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Visualization on J-HMDB dataset. We visualize our OpenMixer detections (in blue boxes) and ground truth (in yellow boxes) on five base classes (in black font) and five novel classes (in red font). Class names are shortened for brevity. The numbers after class names are…
Figure 8
Figure 8. Figure 8: Visualization on UCF101-24 dataset. We visualize our OpenMixer detections (in blue boxes) and ground truth (in yellow boxes) on five base classes (in black font) and five novel classes (in red font). Class names are shortened for brevity. The numbers after class names …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 67 canonical work pages

  1. [78]

    Open-vocabulary spatio-temporal action detection

    Tao Wu, Shuqiu Ge, Jie Qin, Gangshan Wu, and Limin Wang. Open-vocabulary spatio-temporal action detection. arXiv preprint arXiv:2405.10832, 2024. 3, 8, 15

  2. [1]

    Bridg- ing the gap between object and image-level representations for open-vocabulary detection

    Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shahbaz Khan. Bridg- ing the gap between object and image-level representations for open-vocabulary detection. In NeurIPS, pages 33781– 33794, 2022. 1, 2

  3. [2]

    End- to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- to-end object detection with transformers. In ECCV, pages 213–229, 2020. 2, 5, 6

  4. [3]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, pages 9650–9660, 2021. 4

  5. [4]

    Generic attention- model explainability for interpreting bi-modal and encoder- decoder transformers

    Hila Chefer, Shir Gur, and Lior Wolf. Generic attention- model explainability for interpreting bi-modal and encoder- decoder transformers. In ICCV, pages 397–406, 2021. 4

  6. [5]

    CycleACR: Cycle Modeling of Actor-Context Relations for Video Action Detection

    Lei Chen, Zhan Tong, Yibing Song, Gangshan Wu, and Limin Wang. Cycleacr: Cycle modeling of actor-context relations for video action detection. arXiv preprint arXiv:2303.16118, 2023. 2

  7. [6]

    Efficient video action detection with token dropout and context refinement

    Lei Chen, Zhan Tong, Yibing Song, Gangshan Wu, and Limin Wang. Efficient video action detection with token dropout and context refinement. In ICCV, 2023. 1, 2, 4

  8. [7]

    Watch only once: An end-to-end video action detection framework

    Shoufa Chen, Peize Sun, Enze Xie, Chongjian Ge, Jiannan Wu, Lan Ma, Jiajun Shen, and Ping Luo. Watch only once: An end-to-end video action detection framework. In ICCV, pages 8178–8187, 2021. 1, 2, 3, 4, 6

Show all 91 references
  1. [8]

    Gabriellav2: Towards better generalization in surveillance videos for ac- tion detection

    Ishan Dave, Zacchaeus Scheffer, Akash Kumar, Sarah Shi- raz, Yogesh Singh Rawat, and Mubarak Shah. Gabriellav2: Towards better generalization in surveillance videos for ac- tion detection. In WACV, pages 122–132, 2022. 1

  2. [9]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021. 4

  3. [10]

    Learning to prompt for open-vocabulary ob- ject detection with vision-language model

    Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary ob- ject detection with vision-language model. In CVPR, pages 14084–14093, 2022. 5

  4. [11]

    Holistic interaction transformer network for action detection

    Gueter Josmy Faure, Min-Hung Chen, and Shang-Hong Lai. Holistic interaction transformer network for action detection. In WACV, pages 3340–3350, 2023. 4, 6

  5. [12]

    X3d: Expanding architectures for efficient video recognition

    Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR, pages 203–213, 2020. 2

  6. [13]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pages 6202–6211, 2019. 1, 2

  7. [14]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision , pages 1–15, 2023. 5

  8. [15]

    Adamixer: A fast-converging query-based object detector

    Ziteng Gao, Limin Wang, Bing Han, and Sheng Guo. Adamixer: A fast-converging query-based object detector. In CVPR, pages 5364–5373, 2022. 2, 3, 5

  9. [16]

    Video action transformer network

    Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zis- serman. Video action transformer network. In CVPR, June

  10. [17]

    Ava: A video dataset of spatio-temporally localized atomic visual actions

    Chunhui Gu, Chen Sun, David A Ross, Carl V ondrick, Car- oline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, 2018. 8

  11. [18]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In ICCV, pages 2961–2969, 2017. 4, 6, 7

  12. [19]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In ICCV, pages 2961–2969, 2017. 13, 14

  13. [20]

    Interaction-aware prompting for zero-shot spatio-temporal action detection

    Wei-Jhe Huang, Jheng-Hsien Yeh, Min-Hung Chen, Gueter Josmy Faure, and Shang-Hong Lai. Interaction-aware prompting for zero-shot spatio-temporal action detection. In ICCV Workshop, pages 284–293, 2023. 3, 4, 5, 6, 7

  14. [21]

    Towards understanding ac- tion recognition

    Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J Black. Towards understanding ac- tion recognition. In ICCV, pages 3192–3199, 2013. 6

  15. [22]

    Prompting visual-language models for efficient video understanding

    Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, pages 105–124, 2022. 2, 3, 5, 6

  16. [23]

    Prompting visual-language models for efficient video understanding

    Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, pages 105–124, 2022. 13

  17. [24]

    Action tubelet detector for spatio- temporal action localization

    Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari, and Cordelia Schmid. Action tubelet detector for spatio- temporal action localization. In ICCV, 2017. 1, 3

  18. [25]

    Region- aware pretraining for open-vocabulary object detection with vision transformers

    Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region- aware pretraining for open-vocabulary object detection with vision transformers. In CVPR, pages 11144–11154, 2023. 1, 2, 3

  19. [26]

    You only watch once: A unified cnn architecture for real- time spatiotemporal action localization

    Okan K ¨op¨ukl¨u, Xiangyu Wei, and Gerhard Rigoll. You only watch once: A unified cnn architecture for real- time spatiotemporal action localization. arXiv preprint arXiv:1911.06644, 2019. 1, 2, 6, 7

  20. [27]

    F-vlm: Open-vocabulary object detection upon frozen vision and language models

    Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vlm: Open-vocabulary object detection upon frozen vision and language models. In ICLR, 2022. 1, 3

  21. [28]

    Multisports: A multi-person video dataset of spatio-temporally localized sports actions

    Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gang- shan Wu, and Limin Wang. Multisports: A multi-person video dataset of spatio-temporally localized sports actions. In ICCV, pages 13536–13545, 2021. 1 9

  22. [29]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. In ECCV, pages 280–296, 2022. 13

  23. [31]

    Clip surgery for better explainability with enhancement in open- vocabulary tasks

    Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open- vocabulary tasks. arXiv preprint arXiv:2304.05653 , 2023. 13

  24. [33]

    Exploring visual interpretability for con- trastive language-image pre-training

    Yi Li, Hualiang Wang, Yiqun Duan, Hang Xu, and Xi- aomeng Li. Exploring visual interpretability for con- trastive language-image pre-training. arXiv preprint arXiv:2209.07046, 2022. 13

  25. [34]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, pages 7061–7070, 2023. 1

  26. [35]

    Learning object-language alignments for open-vocabulary object de- tection

    Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gho- lamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object de- tection. In ICLR, 2022. 1, 3

  27. [36]

    Frozen clip models are efficient video learners

    Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. InECCV, pages 388–404, 2022. 3

  28. [37]

    Revisiting temporal modeling for clip-based image-to-video knowledge transferring

    Ruyang Liu, Jingjia Huang, Ge Li, Jiashi Feng, Xinglong Wu, and Thomas H Li. Revisiting temporal modeling for clip-based image-to-video knowledge transferring. InCVPR, pages 6555–6564, 2023. 2, 3

  29. [38]

    Revisiting temporal modeling for clip-based image-to-video knowledge transferring

    Ruyang Liu, Jingjia Huang, Ge Li, Jiashi Feng, Xinglong Wu, and Thomas H Li. Revisiting temporal modeling for clip-based image-to-video knowledge transferring. InCVPR, pages 6555–6564, 2023. 13

  30. [40]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 13, 14

  31. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018. 6

  32. [42]

    Verbs in action: Improv- ing verb understanding in video-language models

    Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improv- ing verb understanding in video-language models. In ICCV, pages 15579–15591, 2023. 2

  33. [43]

    Zero-shot temporal action detection via vision-language prompting

    Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xi- ang. Zero-shot temporal action detection via vision-language prompting. In ECCV, pages 681–697, 2022. 2, 3, 5, 6

  34. [44]

    Zero-shot temporal action detection via vision-language prompting

    Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xi- ang. Zero-shot temporal action detection via vision-language prompting. In ECCV, pages 681–697, 2022. 13

  35. [45]

    Expanding language-image pretrained models for gen- eral video recognition

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for gen- eral video recognition. In ECCV, pages 1–18, 2022. 2

  36. [46]

    Expanding language-image pretrained models for gen- eral video recognition

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for gen- eral video recognition. In ECCV, pages 1–18, 2022. 3

  37. [48]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 13

  38. [49]

    Actor-context-actor relation net- work for spatio-temporal action localization

    Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, and Hongsheng Li. Actor-context-actor relation net- work for spatio-temporal action localization. InCVPR, pages 464–474, 2021. 1, 2

  39. [50]

    St-adapter: Parameter-efficient image-to-video transfer learning

    Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hong- sheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. In NeurIPS, pages 26462–26477, 2022. 5

  40. [51]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763, 2021. 1, 2, 4, 6

  41. [52]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763, 2021. 13, 14

  42. [53]

    Fine-tuned clip models are efficient video learners

    Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In CVPR, pages 6545–6554, 2023. 2, 3

  43. [55]

    Open-vocabulary temporal action detection with off-the-shelf image-text features

    Vivek Rathod, Bryan Seybold, Sudheendra Vijaya- narasimhan, Austin Myers, Xiuye Gu, Vighnesh Birodkar, and David A Ross. Open-vocabulary temporal action detection with off-the-shelf image-text features. In BMVC,

  44. [56]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015. 3

  45. [57]

    Generalized in- tersection over union: A metric and a loss for bounding box regression

    Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized in- tersection over union: A metric and a loss for bounding box regression. In CVPR, pages 658–666, 2019. 5

  46. [58]

    Sparse DETR: Efficient end-to-end object detec- tion with learnable sparsity

    Byungseok Roh, JaeWoong Shin, Wuhyun Shin, and Sae- hoon Kim. Sparse DETR: Efficient end-to-end object detec- tion with learnable sparsity. In ICLR, 2022. 2, 5 10

  47. [59]

    Hi- era: A hierarchical vision transformer without the bells-and- whistles

    Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hi- era: A hierarchical vision transformer without the bells-and- whistles. In ICML, 2023. 2

  48. [60]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618–626, 2017. 4

  49. [61]

    Road: The road event awareness dataset for autonomous driving

    Gurkirt Singh, Stephen Akrigg, Manuele Di Maio, Valentina Fontana, Reza Javanmard Alitappeh, Salman Khan, Suman Saha, Kossar Jeddisaravi, Farzad Yousefi, Jacob Culley, Tom Nicholson, Jordan Omokeowa, Stanislao Grazioso, Andrew Bradley, Giuseppe Di Gironimo, and Fabio Cuzzolin....

  50. [62]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. In CRCV-TR-12-01, 2012. 6

  51. [63]

    Actor-centric relation network

    Chen Sun, Abhinav Shrivastava, Carl V ondrick, Kevin Mur- phy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric relation network. In ECCV, pages 318–334, 2018. 1, 2, 3

  52. [64]

    Relational action forecasting

    Chen Sun, Abhinav Shrivastava, Carl V ondrick, Rahul Suk- thankar, Kevin Murphy, and Cordelia Schmid. Relational action forecasting. In CVPR, 2019. 1

  53. [65]

    Sparse r-cnn: End-to-end object detec- tion with learnable proposals

    Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detec- tion with learnable proposals. InCVPR, pages 14454–14463,

  54. [66]

    Asynchronous interaction aggregation for action detection

    Jiajun Tang, Jin Xia, Xinzhi Mu, Bo Pang, and Cewu Lu. Asynchronous interaction aggregation for action detection. In ECCV, pages 71–87, 2020. 2

  55. [67]

    Mlp-mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lu- cas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. In NeurIPS, pages 24261–24272, 2021. 5

  56. [68]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 3, 5

  57. [70]

    Actionclip: A new paradigm for video action recognition

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021. 13

  58. [71]

    Vita-clip: Video and text adaptive clip via multimodal prompting

    Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fa- had Shahbaz Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. InCVPR, pages 23034–23044, 2023. 3, 5

  59. [72]

    Open-vclip: Transforming clip to an open-vocabulary video model via interpolated weight optimization

    Zejia Weng, Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Open-vclip: Transforming clip to an open-vocabulary video model via interpolated weight optimization. In ICML,

  60. [73]

    Long-term feature banks for detailed video understanding

    Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaim- ing He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In CVPR,

  61. [74]

    Context-aware rcnn: A baseline for ac- tion detection in videos

    Jianchao Wu, Zhanghui Kuang, Limin Wang, Wayne Zhang, and Gangshan Wu. Context-aware rcnn: A baseline for ac- tion detection in videos. In ECCV, pages 440–456, 2020. 1

  62. [75]

    Towards open vocab- ulary learning: A survey

    Jianzong Wu, Xiangtai Li, Shilin Xu Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernard Ghanem, et al. Towards open vocab- ulary learning: A survey. arXiv preprint arXiv:2306.15880,

  63. [76]

    Stmixer: A one-stage sparse action detector

    Tao Wu, Mengqi Cao, Ziteng Gao, Gangshan Wu, and Limin Wang. Stmixer: A one-stage sparse action detector. In CVPR, pages 14720–14729, 2023. 1, 2, 3, 4, 5, 6, 8

  64. [77]

    Stmixer: A one-stage sparse action detector

    Tao Wu, Mengqi Cao, Ziteng Gao, Gangshan Wu, and Limin Wang. Stmixer: A one-stage sparse action detector. In CVPR, pages 14720–14729, 2023. 13, 14

  65. [79]

    Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching

    Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In CVPR, pages 7031– 7040, 2023. 1, 2, 3, 5

  66. [80]

    Clip-vip: Adapting pre- trained image-text model to video-language representation alignment

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre- trained image-text model to video-language representation alignment. In ICLR, 2022. 6

  67. [81]

    Clip-vip: Adapting pre- trained image-text model to video-language representation alignment

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre- trained image-text model to video-language representation alignment. In ICLR, 2022. 13, 14

  68. [82]

    Unloc: A unified framework for video localization tasks

    Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid. Unloc: A unified framework for video localization tasks. In ICCV, pages 13623–13633, 2023. 2

  69. [83]

    Detecting human actions in surveillance videos

    Ming Yang, Shuiwang Ji, Wei Xu, Jinjun Wang, Fengjun Lv, Kai Yu, Yihong Gong, Mert Dikmen, Dennis J Lin, and Thomas S Huang. Detecting human actions in surveillance videos. In TRECVID, 2009. 1

  70. [84]

    Contextualized spatio-temporal contrastive learning with self-supervision

    Liangzhe Yuan, Rui Qian, Yin Cui, Boqing Gong, Flo- rian Schroff, Ming-Hsuan Yang, Hartwig Adam, and Ting Liu. Contextualized spatio-temporal contrastive learning with self-supervision. In CVPR, pages 13977–13986, 2022. 4

  71. [85]

    Vision-based garbage dumping action detection for real-world surveillance platform.ETRI Journal, 41(4):494–505, 2019

    Kimin Yun, Yongjin Kwon, Sungchan Oh, Jinyoung Moon, and Jongyoul Park. Vision-based garbage dumping action detection for real-world surveillance platform.ETRI Journal, 41(4):494–505, 2019. 1

  72. [86]

    Open-vocabulary detr with conditional matching

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In ECCV, pages 106–122, 2022. 3, 5

  73. [87]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih- Fu Chang. Open-vocabulary object detection using captions. In CVPR, pages 14393–14402, 2021. 3 11

  74. [88]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih- Fu Chang. Open-vocabulary object detection using captions. In CVPR, pages 14393–14402, 2021. 14

  75. [89]

    From recognition to cognition: Visual commonsense reason- ing

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reason- ing. In CVPR, 2019. 1

  76. [90]

    Vinvl: Revisiting visual representations in vision-language models

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In CVPR, pages 5579–5588, 2021. 1

  77. [91]

    Tuber: Tubelet transformer for video action detection

    Jiaojiao Zhao, Yanyi Zhang, Xinyu Li, Hao Chen, Bing Shuai, Mingze Xu, Chunhui Liu, Kaustav Kundu, Yuanjun Xiong, Davide Modolo, et al. Tuber: Tubelet transformer for video action detection. In CVPR, pages 13598–13607, 2022. 1, 2, 3, 4, 6

  78. [92]

    Mrsn: Multi-relation support network for video action de- tection

    Yin-Dong Zheng, Guo Chen, Minglei Yuan, and Tong Lu. Mrsn: Multi-relation support network for video action de- tection. arXiv preprint arXiv:2304.11975, 2023. 2

  79. [94]

    Regionclip: Region-based language-image pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In CVPR, pages 16793–16803,

  80. [95]

    Learning deep features for discrimi- native localization

    Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discrimi- native localization. In CVPR, pages 2921–2929, 2016. 4

  81. [96]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  82. [97]

    Zegclip: Towards adapting clip for zero-shot se- mantic segmentation

    Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot se- mantic segmentation. In CVPR, pages 11175–11185, 2023. 1

  83. [98]

    For the action type {CLS}, what are the visual descriptions? Please respond with a list of 16 short sentences

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. InICCV, pages 15659–15669, 2023. 5 12 Appendix A. Prompts for Query-Text Alignment To generate text prompts for each action category, we send a request to GPT [48] by using...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.