Pith. sign in

REVIEW 3 major objections 5 minor 54 references

JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read JoVALE detects human actions in video by adaptively fusing audio, visual, and scene-descriptive language features, achieving 40.1% mAP on AVA.

desk verdict Solid multi-modal VAD paper whose 'language' stream is actually image-encoder features — the scene-descriptive gain is real but the linguistic interpretation is unproven. read the letter →

arxiv 2412.13708 v2 pith:KEO4V4B4 submitted 2024-12-18 cs.CV

Taein Son , Soo Won Seo , Jisong Kim , Seok Hwan Lee , Jun Won Choi This is my paper · ORCID
classification cs.CV
keywords videoactiondetectionmulti-modalfusionaudio-visuallearningscenedescriptionimagecaptioningtransformeractor-centricattentionAVAdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces JoVALE, a video action detection method that combines visual frames, audio, and scene descriptions produced by a frozen image-captioning model. The authors argue that fusing these three streams in an actor-centric way—letting each detected person query the multimodal context—allows the model to pick out the cues relevant to each action. JoVALE reports 40.1% mAP on AVA, a 2.4-point improvement over the previous best method, and also reaches new state-of-the-art scores on UCF101-24 and JHMDB51-21. If correct, the result suggests that scene knowledge from pre-trained vision-language models can be used as a cheap auxiliary signal for spatio-temporal action detection.

What carries the argument

The central mechanism is the Actor-centric Multi-modal Fusion Network (AMFN), a Transformer stack that maintains separate action embeddings for audio, visual, and scene-descriptive modalities, all initialized from the same actor proposals. Each layer first applies Multi-modal Feature Encoding (MFE), which pools action embeddings across actors at each time step, runs self-attention over time to form Temporal Bottleneck Features, and jointly encodes those bottlenecks with the modality's context embeddings. Then Multi-modal Feature Aggregation (MFA) temporally aligns the three updated action embeddings and fuses them with per-actor, per-modality weights from an adaptive gated network (a two-layer MLP plus sigmoid). The gated weights allow the model to selectively trust the modality that carries the action-relevant cue for each person.

What would settle it

Replace the BLIP scene-descriptive features in JoVALE with features of the same shape taken from a randomly initialized image encoder, keeping everything else fixed; if AVA mAP does not fall clearly below the reported 34.0 (with audio) and 32.7 (without audio) figures, the scene modality is not doing the semantic work the paper claims.

Watch

Extended reading notes

Core claim

JoVALE is the first video action detection method to combine audio, visual, and scene-descriptive features, where the scene features come from the BLIP image captioning model applied to sampled frames. The core finding is that these three modalities should be fused actor-centrically: each actor proposal carries its own audio, visual, and scene action embeddings, which are refined through a Transformer. A Multi-modal Feature Encoding module uses temporal bottleneck features to compress actor information and keep the computation manageable, and a Multi-modal Feature Aggregation module aligns the temporal streams and combines them with per-actor adaptive weights. This design reaches 40.1% mAP on AVA v2.2, outperforming the previous best method by 2.4 points, and the visual-plus-scene variant alone achieves 84.9% and 91.0% mAP on UCF101-24 and JHMDB51-21, where most clips have no audio.

Load-bearing premise

The scene-descriptive features extracted from the frozen BLIP captioner carry information that is complementary to the visual and audio streams rather than merely re-encoding the same appearance.

Editorial extensions

If this is right

  • JoVALE sets a new state of the art on AVA v2.2 at 40.1% mAP, a 2.4-point gain over the previous best, indicating that actor-centric multimodal fusion is a productive direction for spatio-temporal action detection.
  • The video-plus-scene variant alone reaches 84.9% and 91.0% mAP on UCF101-24 and JHMDB51-21, where over 80% of clips lack audio, suggesting scene-descriptive features are useful even when audio is absent.
  • Ablations show that adding scene-descriptive features to video raises mAP from 28.0 to 32.7, and adding audio on top raises it to 34.0, so each modality contributes a measurable increment.
  • Adaptive gated fusion in MFA beats uniform weighted fusion and late score fusion by 2.1 and 4.6 mAP points, so per-actor adaptive weighting is a key component of the gain.
  • The temporal bottleneck MFE design achieves 34.0 mAP at 25.4 GFLOPs, outperforming joint space-time attention (33.8 mAP at 41.7 GFLOPs), showing the bottleneck structure improves both accuracy and efficiency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not separate whether BLIP features help by supplying object and scene priors or by regularizing the visual stream; a control that freezes the scene stream's gradients or shuffles captions across frames could disentangle these effects.
  • Because the BLIP captioner is frozen and frame-based, its scene descriptors are likely redundant across time; a temporal or video-level captioning model might yield further gains on AVA.
  • The strong results on silent-heavy datasets suggest scene-descriptive language features could serve as a drop-in signal for video action detection in audio-corrupted or silent surveillance footage.
  • The per-actor gated fusion weights are interpretable; visualizing which modality each actor attends to could provide an explanation tool for action detection errors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes JoVALE, a two-stage video action detection model that extracts per-actor proposals with an off-the-shelf person detector and refines them through an Actor-centric Multi-modal Fusion Network. The model fuses three streams: video backbone features, audio spectrogram features, and what the paper calls scene-descriptive features obtained from the image encoder of the BLIP captioning model. The authors evaluate on AVA, UCF101-24, and JHMDB51-21, reporting state-of-the-art mAP on all three datasets, with the marquee result being 40.1% mAP on AVA, 2.4 points above EVAD. The paper includes ablations of modality combinations, fusion strategies, feature-encoding structures, and the gating mechanism, and it links to public code.

Significance. If the reported numbers are reproducible, the paper demonstrates a useful recipe: an actor-centric fusion of multiple frozen or pretrained encoders can improve spatio-temporal action detection on AVA. The ablation suite in Tables 4-7 is a genuine strength, because it isolates the contributions of the video, audio, and scene-descriptive streams and compares several fusion and encoding designs. Reporting GFLOPs in Table 3 also helps the reader judge efficiency. The significance is tempered, however, by two issues: the 'language context' interpretation is not supported by the architecture (the model uses BLIP image-encoder features, not caption text), and the empirical margins are presented without variance estimates. The paper is likely to be a useful benchmark contribution, but its headline novelty claim needs either additional evidence or a careful reframing.

major comments (3)
  1. [Method, Scene-Descriptive Embeddings; Table 4] The central claim that JoVALE uses 'scene-descriptive language context' is not supported by the architecture as described. In the Scene-Descriptive Embeddings paragraph, the scene stream is obtained by applying BLIP's image encoder to sampled frames and linearly projecting the resulting feature maps; the caption text decoder is not used during training or inference. These features are therefore a second RGB appearance stream, not language or caption text. The +4.7 mAP gain from adding this stream (Table 4: video 28.0 vs video+scene 32.7) could be caused by visual ensembling with a differently trained image encoder. To support the language-context claim, please add a control that replaces BLIP's image encoder with an image-only frozen encoder such as an ImageNet-pretrained ViT-B, and ideally a variant that consumes actual caption tokens or text features. If the gain persists with the image-only control, the contribution should be reframed as a two-visual-stream ensemble rather than a language-based method.
  2. [Experiments, Tables 1-4] No measure of training stochasticity is reported. All mAP values are single numbers, and the headline improvement on AVA is a 2.4 mAP margin over EVAD (Table 1), while in Table 4 video+audio improves over video alone by only 0.6 mAP (28.0 vs 28.6). Without variance estimates, multiple seeds, or a significance test, it is difficult to judge whether the reported margins are meaningful. Please report means and standard deviations over at least three seeds, or clearly state and justify why a single run is the accepted practice for these benchmarks.
  3. [Main Results, UCF101-24 and JHMDB51-21; Table 2] The paper states that over 80% of clips in UCF101-24 and JHMDB51-21 lack audio and that the reported results on these datasets use only visual and scene-descriptive features. Consequently, the audiovisual fusion claim is evaluated only on AVA, and the strong results on UCF101-24 and JHMDB51-21 cannot be attributed to the audio stream. The authors should state this limitation in the abstract or conclusions and temper the general 'audiovisual' wording, or add an experiment on the subset of clips that do contain audio.
minor comments (5)
  1. [References / Temporal Alignment] The paper cites Cooper (2019) for temporal alignment and resizing of action embeddings, but the cited entry is an unrelated cochlear-implant article. Please replace it with a proper reference for temporal interpolation or resizing of sequence features.
  2. [Related Work heading] The heading 'Muti-modal Video Action Detection' contains a typo; it should be 'Multi-modal'.
  3. [Figure 1] Figure 1 shows image captions and a text decoder, but the final system uses only image-encoder features. The caption should clarify that the captioning path is used only to motivate or pretrain the encoder and is not part of the inference pipeline.
  4. [Table 3 and surrounding text] The text says the increased computational cost is 'within a reasonable range,' yet JoVALE at 495 GFLOPs is roughly double EVAD's 243 GFLOPs. Please either report actual wall-clock latency or discuss the trade-off in more concrete terms.
  5. [Abstract / Terminology] The abstract describes 'scene descriptive context sourced from image captioning models.' Since the implementation uses image-encoder features rather than generated text, this wording should be aligned with the method to avoid overclaiming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: JoVALE's results come from held-out benchmark evaluation and ablations, not from fitting a parameter to the claimed outcome.

full rationale

JoVALE is an empirical benchmark paper. The model is trained on standard action-detection splits and evaluated on held-out validation/test sets, and the modality ablations in Table 4 are separate training runs measuring held-out mAP. No fitted parameter is renamed as a prediction, and no equation defines a claimed result in terms of its own input. The scene-descriptive stream is built from BLIP image-encoder feature maps that are linearly projected rather than decoded captions, which raises a legitimate question about whether the 4.7 mAP gain over video-only in Table 4 is attributable to language-derived scene context or merely to ensembling a second visual encoder. However, that is an empirical interpretation and ablation-design concern, not a circularity: the paper does not assume the language contribution in order to derive it. The only self-citation (JARViS, Lee et al. 2024, by overlapping authors) appears in related work and is not load-bearing for the central claims. Accordingly, the derivation chain is self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method relies on standard supervised training and frozen pre-trained feature extractors. No ad hoc entities are introduced. The main assumptions are the transferability of frozen BLIP features and the reliability of the person detector. Hyperparameters are standard design choices, though several are not swept or justified.

free parameters (4)
  • Number of AMFN layers (L) = 6
    Chosen by hand; not swept, no sensitivity analysis reported.
  • Transformer embedding dimension (D) = 256
    Standard choice; not justified against alternatives.
  • Actor proposals per frame (K) = 15
    Top-scoring person boxes retained; no analysis of impact on performance.
  • Scene frames sampled (Ts) = 4
    Uniformly sampled frames fed to BLIP; no ablation on this value.
assumptions (3)
  • domain assumption Pre-trained feature extractors (SlowFast, VideoMAE-ViT, BLIP) provide generalizable representations for video action detection.
    The video, audio, and scene encoders are frozen or fine-tuned on target data; the method does not learn these features from scratch. The paper relies on their transferability.
  • domain assumption The Faster R-CNN person detector produces accurate actor proposals across all test videos.
    All action classifications are conditioned on these boxes; if the detector misses an actor, the method cannot recover. The paper fine-tunes the detector but does not analyze failure cases.
  • standard math Standard deep learning components (sigmoid focal loss, AdamW, Transformer layers) behave as described in the cited literature.
    The paper relies on these off-the-shelf techniques without re-derivation or new theoretical guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts." pith.science (2026). https://pith.science/paper/KEO4V4B4

@misc{pith2026241213708,
  author       = {Pith},
  title        = {Pith review of: JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KEO4V4B4}},
  note         = {Machine review of arXiv:2412.13708}
}
read the original abstract

Video Action Detection (VAD) entails localizing and categorizing action instances within videos, which inherently consist of diverse information sources such as audio, visual cues, and surrounding scene contexts. Leveraging this multi-modal information effectively for VAD poses a significant challenge, as the model must identify action-relevant cues with precision. In this study, we introduce a novel multi-modal VAD architecture, referred to as the Joint Actor-centric Visual, Audio, Language Encoder (JoVALE). JoVALE is the first VAD method to integrate audio and visual features with scene descriptive context sourced from large-capacity image captioning models. At the heart of JoVALE is the actor-centric aggregation of audio, visual, and scene descriptive information, enabling adaptive integration of crucial features for recognizing each actor's actions. We have developed a Transformer-based architecture, the Actor-centric Multi-modal Fusion Network, specifically designed to capture the dynamic interactions among actors and their multi-modal contexts. Our evaluation on three prominent VAD benchmarks, including AVA, UCF101-24, and JHMDB51-21, demonstrates that incorporating multi-modal information significantly enhances performance, setting new state-of-the-art performances in the field.

Figures

Figures reproduced from arXiv: 2412.13708 by the authors.

Figure 1
Figure 1. Overview of JoVALE: (top-left) The proposed JoVALE integrates audio, visual, and scene-descriptive features using [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Structure of AMFN: Three independent MFEs en [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Structure of Multi-modal Feature Aggregation. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Different multi-modal fusion strategies: The symbol [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of activation maps: The left side displays heatmaps when using audio, visual, and scene-descriptive [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 29 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Arnab, A.; Dehghani, M.; Heigold, G.; Sun, C.; Lu c i \'c , M.; and Schmid, C. 2021. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, 6836--6846

    2021

  4. [4]

    Bertasius, G.; Wang, H.; and Torresani, L. 2021. Is space-time attention all you need for video understanding? In ICML, volume 2, 4

    2021

  5. [5]

    Carreira, J.; Noland, E.; Hillier, C.; and Zisserman, A. 2019. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987

    arXiv 2019

  6. [6]

    Changpinyo, S.; Sharma, P.; Ding, N.; and Soricut, R. 2021. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 3558--3568

    work page 2021

  7. [7]

    Chen, L.; Tong, Z.; Song, Y.; Wu, G.; and Wang, L. 2023. Efficient video action detection with token dropout and context refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10388--10399

    work page 2023

  8. [8]

    Chen, S.; Sun, P.; Xie, E.; Ge, C.; Wu, J.; Ma, L.; Shen, J.; and Luo, P. 2021. Watch only once: An end-to-end video action detection framework. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 8178--8187

    work page 2021

Show all 54 references
  1. [9]

    Cooper, A. 2019. Hear me out: hearing each other for the first time: the implications of cochlear implant activation. Missouri medicine, 116(6): 469

    2019

  2. [10]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  3. [11]

    J.; Chen, M.-H.; and Lai, S.-H

    Faure, G. J.; Chen, M.-H.; and Lai, S.-H. 2023. Holistic interaction transformer network for action detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 3340--3350

    2023

  4. [12]

    Feichtenhofer, C.; Fan, H.; Malik, J.; and He, K. 2019. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 6202--6211

    2019

  5. [13]

    Gao, R.; Oh, T.-H.; Grauman, K.; and Torresani, L. 2020. Listen to look: Action recognition by previewing audio. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10457--10467

    2020

  6. [14]

    T.; Lucic, M.; Schmid, C.; and Arnab, A

    Georgescu, M.-I.; Fonseca, E.; Ionescu, R. T.; Lucic, M.; Schmid, C.; and Arnab, A. 2023. Audiovisual masked autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 16144--16154

    2023

  7. [15]

    Girdhar, R.; Carreira, J.; Doersch, C.; and Zisserman, A. 2019. Video action transformer network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 244--253

    2019

  8. [16]

    Gkioxari, G.; and Malik, J. 2015. Finding action tubes. In Proceedings of the IEEE conference on computer vision and pattern recognition, 759--768

    2015

  9. [17]

    Gong, Y.; Chung, Y.-A.; and Glass, J. 2021. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778

    2021 arXiv

  10. [18]

    H.; Harwath, D.; Karlinsky, L.; Kuehne, H.; and Glass, J

    Gong, Y.; Rouditchenko, A.; Liu, A. H.; Harwath, D.; Karlinsky, L.; Kuehne, H.; and Glass, J. 2022. Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839

    2022 arXiv

  11. [19]

    A.; Vondrick, C.; Pantofaru, C.; Li, Y.; Vijayanarasimhan, S.; Toderici, G.; Ricco, S.; Sukthankar, R.; et al

    Gu, C.; Sun, C.; Ross, D. A.; Vondrick, C.; Pantofaru, C.; Li, Y.; Vijayanarasimhan, S.; Toderici, G.; Ricco, S.; Sukthankar, R.; et al. 2018. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEEE Conference on Computer Vision an...

    2018

  12. [20]

    He, K.; Chen, X.; Xie, S.; Li, Y.; Doll \'a r, P.; and Girshick, R. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 16000--16009

    2022

  13. [21]

    Huang, P.-Y.; Sharma, V.; Xu, H.; Ryali, C.; Li, Y.; Li, S.-W.; Ghosh, G.; Malik, J.; Feichtenhofer, C.; et al. 2024. Mavil: Masked audio-video learners. Advances in Neural Information Processing Systems, 36

    2024

  14. [22]

    Jhuang, H.; Gall, J.; Zuffi, S.; Schmid, C.; and Black, M. J. 2013. Towards understanding action recognition. In Proceedings of the IEEE International Conference on Computer Vision, 3192--3199

    2013

  15. [23]

    Kay, W.; Carreira, J.; Simonyan, K.; Zhang, B.; Hillier, C.; Vijayanarasimhan, S.; Viola, F.; Green, T.; Back, T.; Natsev, P.; et al. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950

    2017 arXiv

  16. [24]

    Kazakos, E.; Nagrani, A.; Zisserman, A.; and Damen, D. 2019. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the IEEE/CVF international conference on computer vision, 5492--5501

    2019

  17. [25]

    Kim, J.; Koh, J.; Kim, Y.; Choi, J.; Hwang, Y.; and Choi, J. W. 2018. Robust deep multi-modal learning based on gated information fusion network. In Asian Conference on Computer Vision, 90--106. Springer

    2018

  18. [26]

    o p \"u kl \

    K \"o p \"u kl \"u , O.; Wei, X.; and Rigoll, G. 2019. You only watch once: A unified cnn architecture for real-time spatiotemporal action localization. arXiv preprint arXiv:1911.06644

    2019 arXiv

  19. [27]

    A.; et al

    Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123: 32--73

    2017

  20. [28]

    H.; Son, T.; Seo, S

    Lee, S. H.; Son, T.; Seo, S. W.; Kim, J.; and Choi, J. W. 2024. JARViS: Detecting actions in video using unified actor-scene context relation modeling. Neurocomputing, 610: 128616

    2024

  21. [29]

    Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022 a . Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, 12888--12900. PMLR

    2022

  22. [30]

    Li, Y.; Wu, C.-Y.; Fan, H.; Mangalam, K.; Xiong, B.; Malik, J.; and Feichtenhofer, C. 2022 b . Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4804--4814

    2022

  23. [31]

    Lin, T.-Y.; Doll \'a r, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2117--2125

    2017

  24. [32]

    Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 1...

    2014

  25. [33]

    Nagrani, A.; Yang, S.; Arnab, A.; Jansen, A.; Schmid, C.; and Sun, C. 2021. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems, 34: 14200--14213

    2021

  26. [34]

    Ni, B.; Peng, H.; Chen, M.; Zhang, S.; Meng, G.; Fu, J.; Xiang, S.; and Ling, H. 2022. Expanding language-image pretrained models for general video recognition. In European Conference on Computer Vision, 1--18. Springer

    2022

  27. [35]

    Ordonez, V.; Kulkarni, G.; and Berg, T. 2011. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24

    2011

  28. [36]

    Z.; Liu, Y.; Shao, J.; and Li, H

    Pan, J.; Chen, S.; Shou, M. Z.; Liu, Y.; Shao, J.; and Li, H. 2021. Actor-context-actor relation network for spatio-temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 464--474

    2021

  29. [37]

    S.; Chan, W.; Zhang, Y.; Chiu, C.-C.; Zoph, B.; Cubuk, E

    Park, D. S.; Chan, W.; Zhang, Y.; Chiu, C.-C.; Zoph, B.; Cubuk, E. D.; and Le, Q. V. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779

    2019 arXiv

  30. [38]

    Ren, S.; He, K.; Girshick, R.; and Sun, J. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28

    2015

  31. [39]

    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211--252

    2015

  32. [40]

    H.; and Cuzzolin, F

    Saha, S.; Singh, G.; Sapienza, M.; Torr, P. H.; and Cuzzolin, F. 2016. Deep learning for detecting multiple space-time action tubes in videos. arXiv preprint arXiv:1608.01529

    2016 arXiv

  33. [41]

    Schuhmann, C.; Vencu, R.; Beaumont, R.; Kaczmarczyk, R.; Mullis, C.; Katta, A.; Coombes, T.; Jitsev, J.; and Komatsuzaki, A. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114

    2021 arXiv

  34. [42]

    Shah, A.; Mishra, S.; Bansal, A.; Chen, J.-C.; Chellappa, R.; and Shrivastava, A. 2022. Pose and joint-aware action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 3850--3860

    2022

  35. [43]

    R.; and Shah, M

    Soomro, K.; Zamir, A. R.; and Shah, M. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild . arXiv preprint arXiv:1212.0402

    2012 arXiv

  36. [44]

    Tang, J.; Xia, J.; Mu, X.; Pang, B.; and Lu, C. 2020. Asynchronous interaction aggregation for action detection. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XV 16, 71--87. Springer

    2020

  37. [45]

    Tong, Z.; Song, Y.; Wang, J.; and Wang, L. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 10078--10093

    2022

  38. [46]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  39. [47]

    Wang, L.; Huang, B.; Zhao, Z.; Tong, Z.; He, Y.; Wang, Y.; Wang, Y.; and Qiao, Y. 2023 a . Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14549--14560

    2023

  40. [48]

    Wang, R.; Chen, D.; Wu, Z.; Chen, Y.; Dai, X.; Liu, M.; Yuan, L.; and Jiang, Y.-G. 2023 b . Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2023

  41. [49]

    Wu, C.-Y.; Li, Y.; Mangalam, K.; Fan, H.; Xiong, B.; Malik, J.; and Feichtenhofer, C. 2022. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1...

    2022

  42. [50]

    Wu, T.; Cao, M.; Gao, Z.; Wu, G.; and Wang, L. 2023. Stmixer: A one-stage sparse action detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14720--14729

    2023

  43. [51]

    J.; Grauman, K.; Malik, J.; and Feichtenhofer, C

    Xiao, F.; Lee, Y. J.; Grauman, K.; Malik, J.; and Feichtenhofer, C. 2020. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740

    2020 arXiv

  44. [52]

    Xie, S.; Girshick, R.; Doll \'a r, P.; Tu, Z.; and He, K. 2017. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1492--1500

    2017

  45. [53]

    Zhao, J.; and Snoek, C. G. 2019. Dance with flow: Two-in-one stream action detection. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, 9935--9944

    2019

  46. [54]

    Zhao, J.; Zhang, Y.; Li, X.; Chen, H.; Shuai, B.; Xu, M.; Liu, C.; Kundu, K.; Xiong, Y.; Modolo, D.; et al. 2022. Tuber: Tubelet transformer for video action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13598--13607

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.