Pith. sign in

REVIEW 4 major objections 3 minor 22 references

CLIP-AE: CLIP-assisted Cross-view Audio-Visual Enhancement for Unsupervised Temporal Action Localization

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLIP-AE combines frozen CLIP features, audio, and cross-view self-supervision to set state-of-the-art unsupervised temporal action localization on THUMOS14 and ActivityNet v1.2.

desk verdict Sensible incremental UTAL method with internally consistent gains, but the unsupervised protocol is never specified and the weakly/unsupervised row ambiguity undermines the central claim. read the letter →

arxiv 2505.23524 v2 pith:GZZ7URUP submitted 2025-05-29 cs.CV

Rui Xia , Dan Jiang , Quan Zhang , Ke Zhang , Chun Yuan This is my paper
classification cs.CV
keywords unsupervisedtemporalactionlocalizationCLIPaudio-visualfusioncross-viewlearningself-supervisedTHUMOS14ActivityNetv1.2vision-languagepretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Unsupervised temporal action localization asks a model to find when actions happen in untrimmed video using no boundary or category labels, and the paper claims this task can be pushed much further by adding two annotation-free information sources: CLIP's vision-language features and the video's own audio. The proposed method, CLIP-AE, fuses audio with classification-pretrained visual features through cross-attention, lets CLIP features collaborate with classification features so the model attends to whole action intervals instead of only the most discriminative frames, and trains the fusion with self-supervised decorrelation and instance-discrimination losses. On THUMOS14 it reports 41.7% average mAP versus 40.2% for the UGCT baseline, and on ActivityNet v1.2 it reports 28.9% versus 22.7%. If correct, this means unsupervised localization can get closer to supervised accuracy without extra labeling, using pretrained models and audio that are already available in the video.

What carries the argument

The central machinery is a two-view collaboration between classification pre-trained (CBP) features and vision-language pre-trained (VLP) features, plus an audio-visual cross-attention fusion with dense skip connections. In the cross-view paradigm, CBP and VLP segment features are concatenated and passed through multi-modal attention so each view can borrow the other's temporal coverage; in the fusion module, audio and visual segment features are L2-normalized, multiplied through a learnable cross-correlation matrix, softmaxed into attention weights, and iteratively re-weighted with tanh-gated dense skip connections to preserve each modality's identity. The whole representation is trained by two self-supervised proxy tasks, feature decorrelation (pushing the cross-correlation of each view toward the identity) and cross-view instance discrimination with momentum memory banks, so the enhanced features require no labels.

What would settle it

A direct test would be to extract CLIP's per-frame attention over annotated ground-truth action segments in THUMOS14 and check whether it covers whole intervals better than chance; if it does not, the cross-view collaboration's rationale collapses. A second test would rerun CLIP-AE with the audio stream silenced; the paper attributes 2.4% mAP to audio-visual fusion, so if the mAP does not drop when audio is removed, audio is not the actual source of that gain.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a feature-enhancement pipeline, not a new localization head, is what improves unsupervised temporal action localization. Its claim is that by combining three pretrained views (I3D classification features, CLIP vision-language features, and a VGG-like audio network) through a multi-stage dense cross-attention fusion and a cross-view collaboration module, and by training those modules with self-supervised losses, the localizer attends to complete action intervals and gains sharper contextual boundaries. The reported numbers support this: 41.7% average mAP at IoU 0.1:0.7 on THUMOS14 (UGCT: 40.2%), and 28.9% average mAP on ActivityNet v1.2 (UGCT: 22.7%). The paper's ablation attributes 2.4% of the gain to audio-visual fusion, 3.8% to CLIP cross-view collaboration, and a combined 6.2% over the baseline.

Load-bearing premise

The gain depends on CLIP-style vision-language features naturally covering complete action intervals while classification features only catch discriminative frames, and on the THUMOS14 and ActivityNet v1.2 videos having clean, aligned audio; the paper gives no direct evidence for either premise.

Editorial extensions

If this is right

  • On THUMOS14, CLIP-AE reaches 41.7% average mAP (IoU 0.1:0.7), beating the UGCT unsupervised baseline by 1.5 points and APSL by 6.5 points.
  • On ActivityNet v1.2, the 28.9% average mAP surpasses all unsupervised methods and is competitive with several recent weakly supervised methods, narrowing the gap between unsupervised and weakly supervised localization.
  • The ablations show the two modules are complementary rather than redundant: audio-visual fusion alone adds 2.4% mAP, CLIP cross-view collaboration alone adds 3.8%, and both together add 6.2% over the UGCT baseline.
  • Because all additional features come from frozen pretrained models and the video's own audio, the method scales to untrimmed video collections without extra annotation cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension I would expect to confirm the paper's mechanism: per-class results on THUMOS14 should show the largest CLIP-AE gains on sound-associated action classes such as sports or musical performance, and little or no gain on silent classes; the paper does not report this breakdown.
  • The paper's premise about CLIP temporal attention could be verified independently; if true, the same two-view collaboration could transfer to weakly supervised TAL, dense video captioning, or any task needing complete action-interval coverage.
  • The ablations add modules cumulatively, so the individual contributions of audio versus CLIP are not isolated; a cleaner attribution would require a two-by-two ablation with each modality toggled independently.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes CLIP-AE, an unsupervised temporal action localization (UTAL) method that combines classification-pretrained (CBP, I3D), vision-language-pretrained (VLP, CLIP ViT-B), and audio (VGG-like on AudioSet) features. The method consists of an audio-visual cross-attention fusion module (CAF), a cross-view collaborative paradigm (CCP) that mixes CBP and VLP features, and a self-supervised cross-view learning module with feature decorrelation and instance discrimination losses. Experiments on THUMOS14 and ActivityNet v1.2 report state-of-the-art UTAL results, e.g., 41.7% average mAP (IoU 0.1:0.7) on THUMOS14 versus 40.2% for UGCT, and 28.9% average mAP on ActivityNet v1.2 versus 22.7% for UGCT. The ablation study shows monotone gains from adding CAF and CCP. The central claim is that combining VLP features, audio, cross-attention fusion, and cross-view learning improves UTAL accuracy beyond existing visual-only methods.

Significance. The paper attacks a meaningful problem—unsupervised temporal action localization—and its direction of using CLIP and audio is a plausible, relatively unexplored extension. The reported numbers are internally consistent and the ablation shows monotone gains for each module, which is a positive signal. The paper is the first to introduce CLIP and audio into UTAL, which is a contribution if the results hold. However, the central claim is currently not verifiable as stated: the unsupervised training protocol is not specified, the weakly supervised and unsupervised rows in Tables 1–2 are not distinguished, no control ablations isolate the proposed fusion mechanisms from simple feature concatenation, and audio data availability/alignment is never discussed. These issues are load-bearing for the claim that the gains come from the proposed modules. The paper would also benefit from variance estimates or significance tests, given the small margins over prior work. The strength of the contribution is therefore conditional on resolving these verification gaps.

major comments (4)
  1. [Section 3.2; Tables 1 and 2] The unsupervised evaluation protocol is not specified. The paper only states 'Unless otherwise stated, the UTAL baseline is UGCT' and lists feature extractors, but does not describe how the self-supervised losses interact with UGCT's iterative clustering/localization loop, how pseudo-labels are generated, what cluster count is used, or how the loss terms in Eq. (9) are weighted. Tables 1 and 2 list CLIP-AE in both the 'Weakly' and 'Unsupervised' blocks with different scores (THUMOS14: 48.0 vs 41.7 average mAP; ActivityNet v1.2: 30.5 vs 28.9), yet no explanation is given of what training setting distinguishes those rows. Since the paper's contribution is specifically an unsupervised method, the reader cannot determine whether the reported improvements arise from the proposed mechanisms or from action-category label information. This must be clarified.
  2. [Sections 1 and 2.2] The motivating premise that 'VLP features often focus on as complete a temporal action interval as possible' is asserted without supporting evidence. The Cross-View Collaborative Paradigm (CCP) is built on this assumption, but the paper provides no analysis of CLIP temporal attention, no visualization of which temporal segments CLIP highlights, and no control experiment that uses CLIP features without the cross-view attention mechanism. Without such evidence, the reported gain from CCP could equally come from feature concatenation or increased feature dimensionality rather than from the supposed completeness of VLP temporal coverage.
  3. [Section 3.4, Table 3] The ablation does not include controls that simply add raw audio features or raw CLIP features to the baseline. Table 3 shows that adding CAF and CCP increases average mAP on ActivityNet v1.2 from 22.7% to 28.9%, but without a configuration that concatenates raw audio features to CBP features (or raw CLIP features) without the proposed cross-attention and cross-view mechanisms, the contribution of the novel fusion designs is not isolated. Please add these controls to verify that the gains are due to the proposed modules rather than to the mere presence of additional features.
  4. [Section 3.1 and 3.2] The paper never reports whether audio tracks exist for THUMOS14 and ActivityNet v1.2, or how audio is aligned with the visual features. Since audio-visual fusion is a core contribution, the validity and reproducibility of the results hinge on audio availability and preprocessing details (e.g., sampling rate, audio feature extractor specifics, handling of videos without audio). This information must be provided for both datasets.
minor comments (3)
  1. [Table 3 caption] The word 'Albaltion' should be 'Ablation'.
  2. [Eq. (8)] The notation in Eq. (8) is inconsistent: the numerator uses z+_VLP and z+_CBP, while the denominator and the following text use x+_VLP and x+_CBP. Please unify the notation and define how positive pairs are constructed for each view.
  3. [Section 2.1] In Eq. (1), the dimensions and indexing of Lambda are not fully clear; the text says 'the l-th column of Lambda corresponds to the correlation between x^l_CBP and L audio segment features,' but the formula Lambda = X_audio^T W X_CBP should be accompanied by explicit dimension statements for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CLIP-AE's self-supervised losses are defined on input features and are not equivalent to the reported mAP.

full rationale

No circular step is present. CLIP-AE's training objectives (Eqs. 7-8) are self-supervised pretext tasks, feature decorrelation and instance discrimination, computed directly from the extracted audio, CBP, and VLP features; they contain no action labels, no fitted pseudo-labels, and no test-set statistics. The reported mAP is therefore an external evaluation metric, not an identity or a re-fit of the training objective. The attention and fusion modules (Eqs. 1-6) transform features with learnable parameters optimized by those self-supervised losses, and nothing in the equations defines the predicted localization output as equivalent to the input features or annotations. The placement of CLIP-AE in both weakly supervised and unsupervised rows of Tables 1-2 is an experimental-protocol ambiguity that affects verifiability and fair comparison, but it does not make the central claim circular. Self-citations (refs 4-5 and 13-16) appear only as background or as examples of deep learning applications and are not load-bearing. The premise that VLP features capture complete temporal action intervals is asserted without direct evidence; this is a correctness and evidence concern, not a circularity concern. The derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on frozen pretrained features, on the untested assumption that CLIP features complete action intervals, and on the UGCT baseline. No new physical or conceptual entities are introduced. Several implementation hyperparameters are either set by hand or unreported.

free parameters (5)
  • temperature tau = 1.0
    Set by hand in Eq. (8) for instance discrimination; no sensitivity analysis is reported.
  • loss weighting = equal weights, implicitly 1.0 and 1.0
    In Eq. (9), L_self = L_de-cor + L_ins-dis, with no ablation over alternative weights.
  • number of cross-attention fusion stages = not reported
    Eq. (3) indexes stages t but the paper never states how many stages are used; this affects fusion depth.
  • memory bank size = not reported
    Eq. (8) uses a memory bank {m_i}_{i=1}^N, but N is never specified.
  • feature dimension d_x = not reported
    Section 2.1 maps audio and CBP features to R^{d_x}, but the dimension is not given.
assumptions (5)
  • domain assumption Pre-trained I3D, CLIP ViT-B, and AudioSet VGG features transfer to UTAL without fine-tuning.
    Section 3.2 states the extractors; no fine-tuning or domain adaptation is described. If transfer fails, all results depend on it.
  • domain assumption VLP features capture complete temporal action intervals better than CBP features.
    Section 1 states this as the inspiration for CCP; no evidence is provided. This motivates the central cross-view collaboration.
  • domain assumption Audio is available and temporally aligned for THUMOS14 and ActivityNet v1.2.
    Section 2.1 uses audio features, but the paper does not report missing-audio handling or audio statistics for the two datasets.
  • domain assumption The UGCT baseline supplies reliable pseudo-labels and clusters for UTAL.
    Section 3.2 says the UTAL baseline is UGCT; all ablations start from UGCT, so the method inherits its clustering and pseudo-label quality.
  • domain assumption Self-supervised decorrelation and instance discrimination losses improve representation quality for localization.
    Section 2.3 and Eq. (9) introduce these losses; the paper provides only end-to-end mAP, not a direct analysis of what the learned representations change.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLIP-AE: CLIP-assisted Cross-view Audio-Visual Enhancement for Unsupervised Temporal Action Localization." pith.science (2026). https://pith.science/paper/GZZ7URUP

@misc{pith2026250523524,
  author       = {Pith},
  title        = {Pith review of: CLIP-AE: CLIP-assisted Cross-view Audio-Visual Enhancement for Unsupervised Temporal Action Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZZ7URUP}},
  note         = {Machine review of arXiv:2505.23524}
}
read the original abstract

Temporal Action Localization (TAL) has garnered significant attention in information retrieval. Existing supervised or weakly supervised methods heavily rely on labeled temporal boundaries and action categories, which are labor-intensive and time-consuming. Consequently, unsupervised temporal action localization (UTAL) has gained popularity. However, current methods face two main challenges: 1) Classification pre-trained features overly focus on highly discriminative regions; 2) Solely relying on visual modality information makes it difficult to determine contextual boundaries. To address these issues, we propose a CLIP-assisted cross-view audiovisual enhanced UTAL method. Specifically, we introduce visual language pre-training (VLP) and classification pre-training-based collaborative enhancement to avoid excessive focus on highly discriminative regions; we also incorporate audio perception to provide richer contextual boundary information. Finally, we introduce a self-supervised cross-view learning paradigm to achieve multi-view perceptual enhancement without additional annotations. Extensive experiments on two public datasets demonstrate our model's superiority over several state-of-the-art competitors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 21 canonical work pages

  1. [1]

    Tradi- tional methods primarily focus on identifying the most rele- vant videos based on a given query [1]

    INTRODUCTION With the rapid growth of social media videos, video retrieval has become a popular topic in information retrieval. Tradi- tional methods primarily focus on identifying the most rele- vant videos based on a given query [1]. However, users are often more interested in specific actions within video clips, making temporal action localization (TAL...

    work page

  2. [2]

    Classification pre-trained features overly focus on highly discriminative regions. The features used in UTAL for clus- tering and localization are extracted based on classification pre-training, which often only needs to focus on key frames of action instances for classification tasks, lacking sufficient attention to the complete action interval. 2) Solel...

    work page Pith review arXiv 2025

  3. [3]

    Pre-trained audio, VLP, and CBP feature extractors first extractF audio,F CBP , andF V LP features

    METHOD Our CLIP-assisted cross-view audio-visual enhancement framework is shown in Figure 2. Pre-trained audio, VLP, and CBP feature extractors first extractF audio,F CBP , andF V LP features. These are processed through the Audio-Visual Cross Attention Fusion module to generate fused audio-visual fea- tures. Simultaneously,F CBP andF V LP are input into ...

    work page

  4. [4]

    Datasets THUMOS14consists of 200 validation and 213 test videos across 20 action classes, averaging 15 action segments per video

    EXPERIMENTS 3.1. Datasets THUMOS14consists of 200 validation and 213 test videos across 20 action classes, averaging 15 action segments per video. Validation data is used for training, and test data for evaluation.ActivityNet v1.2includes 4819 training, 2383 validation, and 2480 test videos. The validation set is used as test data, with an average of 1.5 ...

    work page

  5. [5]

    This is the first time CLIP and audio are incorporated into UTAL

    CONCLUSION We propose CLIP-AE, a CLIP-assisted cross-view audio- visual perception learning paradigm, to address unsuper- vised temporal action localization (UTAL). This is the first time CLIP and audio are incorporated into UTAL. Our ap- proach introduces a cross-view collaboration between CBP and VLP, enabling mutual enhancement. We also present a cross...

    work page

  6. [6]

    ACKNOWLEDGEMENT This work was supported by the National Key R&D Program of China (2022YFB4701400/4701402), SSTIC Grant(KJZD2 0230923115106012,KJZD20230923114916032,GJHZ20240 218113604008), Beijing Key Lab of Networked Multimedia and National Natural Science Foundation of China under Grant 62202302

    work page

  7. [7]

    Two-stream consensus network for weakly-supervised temporal action local- ization,

    Yuanhao Zhai, Le Wang, Wei Tang, Qilin Zhang, Jun- song Yuan, and Gang Hua, “Two-stream consensus network for weakly-supervised temporal action local- ization,” inComputer Vision–ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16. Springer, 2020, pp. 37–54

    work page 2020

  8. [8]

    Weakly-supervised temporal action lo- calization by inferring salient snippet-feature,

    Wulian Yun, Mengshi Qi, Chuanming Wang, and Huadong Ma, “Weakly-supervised temporal action lo- calization by inferring salient snippet-feature,” inPro- ceedings of the AAAI Conference on Artificial Intelli- gence, 2024, vol. 38, pp. 6908–6916

    work page 2024

Show all 22 references
  1. [9]

    Two-stream networks for weakly-supervised temporal action local- ization with semantic-aware mechanisms,

    Yu Wang, Yadong Li, and Hongbin Wang, “Two-stream networks for weakly-supervised temporal action local- ization with semantic-aware mechanisms,” inProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18878–18887

    2023

  2. [10]

    Can mllms guide weakly- supervised temporal action localization tasks?,

    Quan Zhang and Yuxin Qi, “Can mllms guide weakly- supervised temporal action localization tasks?,”arXiv preprint arXiv:2411.08466, 2024

    2024 arXiv

  3. [11]

    Rethinking pseudo-label guided learning for weakly supervised temporal action localization from the perspective of noise correction,

    Quan Zhang, Yuxin Qi, Xi Tang, Rui Yuan, Xi Lin, Ke Zhang, and Chun Yuan, “Rethinking pseudo-label guided learning for weakly supervised temporal action localization from the perspective of noise correction,” Proceedings of the AAAI Conference on Artificial Intel- ligence, vol...

    2025

  4. [12]

    Apsl: Action-positive separation learning for unsupervised temporal action localization,

    Yuanyuan Liu, Ning Zhou, Fayong Zhang, Wenbin Wang, Yu Wang, Kejun Liu, and Ziyuan Liu, “Apsl: Action-positive separation learning for unsupervised temporal action localization,”Information Sciences, vol. 630, pp. 206–221, 2023

    2023

  5. [13]

    Learning temporal co-attention models for unsupervised video action localization,

    Guoqiang Gong, Xinghan Wang, Yadong Mu, and Qi Tian, “Learning temporal co-attention models for unsupervised video action localization,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9819–9828

    2020

  6. [14]

    Uncertainty guided collaborative training for weakly supervised and unsupervised temporal action localization,

    Wenfei Yang, Tianzhu Zhang, Yongdong Zhang, and Feng Wu, “Uncertainty guided collaborative training for weakly supervised and unsupervised temporal action localization,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 5252–5267, 2022

    2022

  7. [15]

    Revisiting foreground and background separation in weakly-supervised temporal action local- ization: A clustering-based approach,

    Qinying Liu, Zilei Wang, Shenghai Rong, Junjie Li, and Yixin Zhang, “Revisiting foreground and background separation in weakly-supervised temporal action local- ization: A clustering-based approach,” inProceedings of the IEEE/CVF International Conference on Com- puter Vision, ...

    2023

  8. [16]

    Weakly supervised action localization by sparse temporal pooling network,

    Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han, “Weakly supervised action localization by sparse temporal pooling network,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6752–6761

    2018

  9. [17]

    Weakly-supervised action localization with background modeling,

    Phuc Xuan Nguyen, Deva Ramanan, and Charless C Fowlkes, “Weakly-supervised action localization with background modeling,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 5502–5511

    2019

  10. [18]

    Proposal-based multiple instance learning for weakly-supervised temporal action localization,

    Huan Ren, Wenfei Yang, Tianzhu Zhang, and Yong- dong Zhang, “Proposal-based multiple instance learning for weakly-supervised temporal action localization,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 2394–2404

    2023

  11. [19]

    IMDPrompter: Adapting SAM to image manipulation detection by cross-view au- tomated prompt learning,

    Quan Zhang, Yuxin Qi, Xi Tang, Jinwei Fang, Xi Lin, Ke Zhang, and Chun Yuan, “IMDPrompter: Adapting SAM to image manipulation detection by cross-view au- tomated prompt learning,” inThe Thirteenth Interna- tional Conference on Learning Representations, 2025

    2025

  12. [20]

    Gim: A million-scale benchmark for generative image manipulation detection and localization,

    Yirui Chen, Xudong Huang, Quan Zhang, Wei Li, Mingjian Zhu, Qiangyu Yan, Simiao Li, Hanting Chen, Hailin Hu, Jie Yang, et al., “Gim: A million-scale benchmark for generative image manipulation detection and localization,” inProceedings of the AAAI Confer- ence on Artificial In...

    2025

  13. [21]

    Seeing beyond noise: Joint graph structure evaluation and denoising for mul- timodal recommendation,

    Yuxin Qi, Quan Zhang, Xi Lin, Xiu Su, Jiani Zhu, Jingyu Wang, and Jianhua Li, “Seeing beyond noise: Joint graph structure evaluation and denoising for mul- timodal recommendation,” inProceedings of the AAAI Conference on Artificial Intelligence, 2025, vol. 39, pp. 12461–12469

    2025

  14. [22]

    Distilling semantic priors from sam to effi- cient image restoration models,

    Quan Zhang, Xiaoyu Liu, Wei Li, Hanting Chen, Jun- chao Liu, Jie Hu, Zhiwei Xiong, Chun Yuan, and Yunhe Wang, “Distilling semantic priors from sam to effi- cient image restoration models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVP...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.