Pith. sign in

REVIEW 3 major objections 5 minor 67 references

Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that audio, converted into video pseudo-labels by a large language model, provides a reliable unsupervised supervisory signal that substantially improves test-time adaptation of video classification models.

desk verdict Audio-assisted pseudo-labels are a genuinely new and well-ablated idea for video TTA, but the reported gains depend on a clean-audio oracle and the paper's generality claims outrun the evidence. read the letter →

arxiv 2506.12481 v1 pith:QI36AJ2C submitted 2025-06-14 cs.CV cs.LGcs.SDeess.AS

Runhao Zeng , Qi Deng , Ronghao Zhang , Shuaicheng Niu , Jian Chen , Xiping Hu , Victor C. M. Leung This is my paper · ORCID
classification cs.CVcs.LGcs.SDeess.AS
keywords test-timeadaptationvideoclassificationaudio-assistedpseudo-labelslargelanguagemodelsaudio-visuallearningout-of-distributionrobustnesslabelmapping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that test-time adaptation of video classification models can be substantially improved by listening: instead of relying only on corrupted visual frames for supervisory signals, the method uses the video's own audio, which survives visual corruption largely intact. A pretrained audio classifier produces audio categories, and a large language model translates those categories into the video model's label space, yielding audio-assisted pseudo-labels that supervise adaptation without any extra training. On two existing and two newly built corrupted-video benchmarks the approach beats prior video TTA methods on every corruption tested, for example raising accuracy on AVMIT-C with a TANet backbone from 59.76% to 74.78%. The paper also introduces a flexible adaptation cycle that decides per sample how many update steps to take, which matters because the optimal number of repetitions varies with the sample and the corruption.

What carries the argument

The load-bearing mechanism is the audio-to-video label mapping via an LLM prompt, combined with a flexible adaptation cycle. The mapping module takes the top five audio-class predictions (labels plus probabilities) from a pretrained audio model and the full video label space, and asks the LLM via a five-part prompt (background, task, requirements, examples, inputs) to output the single most relevant video category; this converts an unmatched audio label space into usable video pseudo-labels with no training. The adaptation cycle then decides how many gradient steps each test sample gets, continuing while either the cross-view consistency loss or the feature-alignment loss keeps decreasing or the two temporal views still disagree, capped by a hyperparameter tau. The classification loss with the audio-assisted pseudo-label is combined with these two visual losses (weighted by 0.1 each) into the total adaptation objective.

What would settle it

Corrupt the audio track as well in AVMIT-C—say, with Gaussian noise, silence, or a mismatched soundtrack—and rerun the method with the reported hyperparameters; if the roughly 15-point accuracy advantage over ViTTA does not substantially shrink, the claim that audio provides the supervision would be called into question.

Watch

Extended reading notes

Core claim

The central discovery is that audio can act as a reliable, untrained source of supervision for video test-time adaptation. Given a corrupted test video, the method extracts the audio track, classifies it with an open-source pretrained audio model such as AST, and feeds the top five audio labels with their confidence scores into a large language model prompted with the video label space; the LLM returns the most plausible video category, which becomes the pseudo-label used in the adaptation loss. Because audio and video labels are both semantically rich text, the LLM bridges the label-space gap without any training. To use these labels well, the method adapts each sample repeatedly but stops per sample based on whether the consistency loss or alignment loss is still decreasing and whether different temporal views of the sample disagree in their predictions. The result is consistent gains across backbones, corruptions, and datasets, including datasets where audio-video correlation is weak and pseudo-label accuracy is near chance.

Load-bearing premise

The reported gains rest on the assumption that at test time the audio track stays clean and semantically aligned with the video content while only the visual stream is corrupted; if the audio is also corrupted, missing, or unrelated to the action, the pseudo-labels lose their value.

Editorial extensions

If this is right

  • If the claim holds, any pretrained video classifier can be made more robust to visual corruption at test time without retraining, fine-tuning, or labeled test data.
  • The method provides a template for leveraging other pretrained modalities (for example, text or depth) as test-time supervisors, since the LLM mapping step can in principle align any label vocabulary to the target label space.
  • The flexible adaptation cycle shows that per-sample adaptive iteration counts are more effective than a fixed number of updates, which suggests that one-size-fits-all TTA schedules leave accuracy on the table.
  • Results on UCF101-C, where pseudo-label accuracy is only 9.67%, indicate that even weak or partially noisy audio-derived labels can still contribute when combined with visual consistency and alignment losses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate testable extension is to corrupt the audio stream as well (adding noise, silence, or mismatched soundtracks) and measure how much of the reported gain survives; the paper's motivating scenarios assume audio stays clean, so the gain is likely to shrink as audio corruption increases.
  • The LLM mapping step could be probed directly: one swapped-in test is to replace the LLM with a simple nearest-neighbor embedding match between audio labels and video labels, which would isolate how much of the benefit comes from the LLM's semantic reasoning versus from any label-alignment at all.
  • Since the method reuses ViTTA's consistency and alignment losses, part of the headline gain may come from the interaction of those losses with the audio pseudo-labels rather than from the audio alone; ablations that vary alpha and beta would separate the contributions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces an audio-assisted test-time adaptation (TTA) method for video classifiers. Given a test video, the method extracts audio, classifies it with a pretrained audio model (AST), and uses an LLM prompt to map the top-K audio labels into the video label space, producing audio-assisted pseudo-labels. The video model is then adapted with a classification loss on these pseudo-labels together with cross-view consistency and feature-alignment losses adapted from ViTTA, and a flexible adaptation cycle decides per sample how many update steps to run. The authors construct two new corrupted audio-video datasets (AVE-C and AVMIT-C) and evaluate on UCF101-C and Kinetics-Sounds-C with TANet and TSM backbones, reporting consistent improvements over existing TTA baselines, e.g., 74.78% vs 59.76% on AVMIT-C with TANet.

Significance. If the reported results are robust, the work is a meaningful step: it shows that the often-ignored audio stream can provide free, test-time supervision for video models without retraining, and it contributes two new benchmark datasets. The ablation in Table III cleanly isolates the audio contribution, and the code is publicly available. The practical delayed-update and label-filtering strategies are useful. However, the current evaluation supports the method primarily when the audio stream is clean and the audio model is in-domain; the headline gains are not demonstrated under audio corruption, and the absence of error bars and the per-dataset hyperparameter choices weaken the reliability of the comparisons. The core idea is sound but the scope of the claims needs to be narrowed or further evidenced.

major comments (3)
  1. [Sec. IV-B, Table III] The evaluation corrupts only the visual stream (Sec. IV-B), leaving the audio stream clean. Table III shows that removing L_cls drops AVMIT-C from 74.78% to 59.39%, nearly the ViTTA baseline of 59.76%, so the headline gain is almost entirely attributable to clean audio pseudo-labels. The paper's general claim that audio can serve as a reliable unsupervised supervisory signal for video TTA is therefore only supported under a clean-audio oracle. Please add experiments with corrupted or partially masked audio (e.g., adding Gaussian noise to the spectrogram or replacing audio with background music) and report performance under those conditions; the currently reported 20% pseudo-label availability experiment (Table VI) does not address audio corruption.
  2. [Sec. IV-D, Tables I-II] The learning rate and the maximum number of adaptation cycles tau are set separately for each dataset and backbone (Sec. IV-D), and the choice of tau is justified by observing test performance (e.g., tau=4 on TSM 'as the performance tends to stabilize'). No error bars or multiple seeds are reported. Since TTA methods are sensitive to these hyperparameters, the reported margins may partly reflect test-set selection rather than the method's intrinsic advantage. Please clarify the hyperparameter selection procedure, provide sensitivity analyses over tau and learning rate, and report mean and standard deviation over at least three runs.
  3. [Sec. V-A, Table II] The claim that the method 'consistently improves' performance is too strong given the out-of-domain audio case. On UCF101-C, where the AST model is out-of-domain and pseudo-label accuracy is only 9.67%, the gain over ViTTA is 1.05 points (78.92 vs 77.87) and the paper attributes this gain to the visual consistency and alignment terms rather than to audio. This is consistent with the method, but it means the benefit of audio-assisted pseudo-labels is confined to settings where the audio model is expert in the test-domain audio distribution. The introduction and abstract should qualify the claims accordingly, and the discussion should state the conditions under which audio supervision is expected to help.
minor comments (5)
  1. [Abstract and Contributions] The contribution bullet '27% accuracy improvement' over ViTTA on AVMIT-C is inconsistent with Table I, where 74.78% vs 59.76% is a 15.02-point absolute increase and about 25.1% relative improvement. Please correct this number.
  2. [Throughout] There are several typos and inconsistent dataset names: 'aaudio-video' in Sec. I, 'A VE-C' and 'A VMIT-C' with spacing issues in Tables and text, and 'Kinetic-sound-C' vs 'Kinetics-Sounds-C' in Table II and Sec. IV-A.
  3. [Fig. 5] In Fig. 5, the x-axis label 'Learning Iteration Step' is unclear; it should be 'Adaptation cycle index' or similar.
  4. [Table VI] Table VI uses m% in the caption but the text says 'probability m%'; please unify notation and state how the availability is simulated (e.g., random per sample).
  5. [Sec. V-B] In Sec. V-B, the baseline with alpha=0 is described as 'combining ViTTA with multiple adaptations'; please clarify exactly which losses remain in this baseline, since it may be interpreted as an ablation of L_cls rather than of ViTTA itself.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: audio pseudo-labels are external, adaptation losses are standard, and the flexible cycle is a stopping rule rather than a fitted prediction.

full rationale

The claimed derivation chain is not circular. The audio-assisted pseudo-label y_v is produced by Eq. (3) from an open-source pretrained audio model (AST) and an LLM, given the audio top-k predictions and the video label space; it does not depend on the target video model's parameters or predictions, so optimizing Eq. (4) against it is genuine external supervision rather than self-confirmation. The additional losses L_cons and L_align are borrowed from ViTTA [14] and align test statistics to pre-calculated training statistics, which is a standard self-supervisory regularization, not a renamed version of the reported accuracy. The flexible adaptation cycle condition in Eq. (10) is a per-sample early-stopping rule based on internal loss decreases and view consistency; it does not encode or presuppose the final test accuracy. The main caveats raised by a skeptical reading—clean-audio evaluation, the AVE-C/AudioSet overlap with AST pretraining, low pseudo-label accuracy on UCF101-C, and dataset-specific hyperparameter choices (e.g., tau and learning rates selected per benchmark)—are evaluation and generalization concerns, not circular reductions. No equation in the paper is equivalent to its own output by construction, and no fitted parameter is relabeled as a prediction. The few self-citations (e.g., AME [15] for the UCF101 split choice, ETTA [11] in related work) are not load-bearing for the central claim that external audio cues can improve video TTA. Therefore the paper's derivation is self-contained with respect to its stated inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on two external models, AST and an LLM, plus a set of hyperparameters chosen on the test set. No new physical or architectural entities are introduced; audio-assisted pseudo-labels are a supervision signal, not a new entity. The main free choices are the cycle cap tau, the loss weights, the learning rates, and top-K.

free parameters (4)
  • maximum adaptation cycles tau = 8 (AVMIT-C), 8/4 (AVE-C TANet/TSM), 3 (UCF101-C), 2 (Kinetics-Sounds-C)
    Chosen per dataset and backbone based on observed test performance; Sec IV-D says performance stabilizes when tau exceeds 4 on TSM.
  • loss weights alpha and beta = 0.1
    Set by hand in Sec III-D: we set alpha equals beta equals 0.1 in all the experiments and find that it works well.
  • learning rate = 5e-6 TANet/AVMIT, 1e-6 TSM/AVE, 1e-5 TANet/AVE, 5e-5 TANet/UCF101-Kinetics
    Tuned per dataset and backbone in Sec IV-D; no validation protocol is given.
  • top-K audio predictions = 5
    Prompt module selects the top five audio categories, Sec III-C.
assumptions (4)
  • domain assumption Visual corruptions do not degrade the audio stream
    The paper corrupts only video frames when building AVE-C and AVMIT-C (Sec IV-B) and argues from Sec I that audio remains informative under fog, motion blur, and compression.
  • domain assumption LLM can reliably map audio labels to the video label space from a prompt
    The whole pseudo-label pipeline in Eq. (3) depends on this; the paper gives examples and error analysis (Sec V-F, Table VII) but reports pseudo-label accuracy only on UCF101 (9.67%), not on the new datasets.
  • domain assumption Pretrained AST top-5 labels contain useful audio-semantic information
    AST outputs are the sole audio evidence; this enters in Sec III-C and in Eq. (3).
  • domain assumption Pre-calculated source training statistics are available for feature alignment
    L_align uses mu-hat and sigma-hat from source training data, following ViTTA [14], Eq. (8).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation." pith.science (2026). https://pith.science/paper/QI36AJ2C

@misc{pith2026250612481,
  author       = {Pith},
  title        = {Pith review of: Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QI36AJ2C}},
  note         = {Machine review of arXiv:2506.12481}
}
read the original abstract

Test-time adaptation (TTA) aims to boost the generalization capability of a trained model by conducting self-/unsupervised learning during the testing phase. While most existing TTA methods for video primarily utilize visual supervisory signals, they often overlook the potential contribution of inherent audio data. To address this gap, we propose a novel approach that incorporates audio information into video TTA. Our method capitalizes on the rich semantic content of audio to generate audio-assisted pseudo-labels, a new concept in the context of video TTA. Specifically, we propose an audio-to-video label mapping method by first employing pre-trained audio models to classify audio signals extracted from videos and then mapping the audio-based predictions to video label spaces through large language models, thereby establishing a connection between the audio categories and video labels. To effectively leverage the generated pseudo-labels, we present a flexible adaptation cycle that determines the optimal number of adaptation iterations for each sample, based on changes in loss and consistency across different views. This enables a customized adaptation process for each sample. Experimental results on two widely used datasets (UCF101-C and Kinetics-Sounds-C), as well as on two newly constructed audio-video TTA datasets (AVE-C and AVMIT-C) with various corruption types, demonstrate the superiority of our approach. Our method consistently improves adaptation performance across different video classification models and represents a significant step forward in integrating audio information into video TTA. Code: https://github.com/keikeiqi/Audio-Assisted-TTA.

Figures

Figures reproduced from arXiv: 2506.12481 by the authors.

Figure 1
Figure 1. Existing video test-time adaptation methods rely on visual supervision, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Common scenarios of video disruption include challenging environ [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overall illustration of our proposed method. Given a paired video-audio test sample, we first extract audio labels using a pre-trained audio classification [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative analysis of the results before and after adaptation using audio information. Results on UCF101-C and Kinetics-Sounds-C. We present comparisons using the UCF101-C and Kinetics￾Sounds-C datasets in Table II. On the Kinetics-Sounds-C dataset, our method improv…
Figure 5
Figure 5. Figure 5: Ablation on TTA’s learning iteration steps (i.e., adaptation cycles) on [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Three cases of inaccurate LLM pseudo labels. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Our proposed prompt for pseudo label mapping. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 55 canonical work pages

  1. [1]

    Action-net: Multipath excitation for action recognition,

    Z. Wang, Q. She, and A. Smolic, “Action-net: Multipath excitation for action recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 13 214–13 223. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 13

    work page 2021

  2. [2]

    Spa- tiotemporal self-attention modeling with temporal patch shift for action recognition,

    W. Xiang, C. Li, B. Wang, X. Wei, X.-S. Hua, and L. Zhang, “Spa- tiotemporal self-attention modeling with temporal patch shift for action recognition,” inProceedings of the European Conference on Computer Vision (ECCV), 2022, pp. 627–644

    work page 2022

  3. [3]

    Recurring the transformer for video action recognition,

    J. Yang, X. Dong, L. Liu, C. Zhang, J. Shen, and D. Yu, “Recurring the transformer for video action recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 14 063–14 073

    work page 2022

  4. [4]

    Graph convolutional module for temporal action localization in videos,

    R. Zeng, W. Huang, M. Tan, Y . Rong, P. Zhao, J. Huang, and C. Gan, “Graph convolutional module for temporal action localization in videos,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 6209–6223, 2021

    work page 2021

  5. [5]

    Tent: Fully test-time adaptation by entropy minimization,

    D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell, “Tent: Fully test-time adaptation by entropy minimization,” inProceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  6. [6]

    Memo: Test time robustness via adaptation and augmentation,

    M. Zhang, S. Levine, and C. Finn, “Memo: Test time robustness via adaptation and augmentation,” inAdvances in Neural Information Processing Systems (NeurIPS), 2022, pp. 38 629–38 642

    work page 2022

  7. [7]

    Test-time classifier adjustment module for model-agnostic domain generalization,

    Y . Iwasawa and Y . Matsuo, “Test-time classifier adjustment module for model-agnostic domain generalization,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 2427– 2440

    work page 2021

  8. [8]

    The norm must go on: dynamic unsupervised domain adaptation by normalization,

    M. J. Mirza, J. Micorek, H. Possegger, and H. Bischof, “The norm must go on: dynamic unsupervised domain adaptation by normalization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 14 765–14 775

    work page 2022

Show all 67 references
  1. [10]

    Test-time training with masked autoencoders,

    Y . Gandelsman, Y . Sun, X. Chen, and A. Efros, “Test-time training with masked autoencoders,” inAdvances in Neural Information Processing Systems (NeurIPS), 2022, pp. 29 374–29 385

    2022

  2. [11]

    Efficient test-time model adaptation without forgetting,

    S. Niu, J. Wu, Y . Zhang, Y . Chen, S. Zheng, P. Zhao, and M. Tan, “Efficient test-time model adaptation without forgetting,” inProceedings of the International Conference on Machine Learning (ICML), 2022, pp. 16 888–16 905

    2022

  3. [12]

    Test- time training with self-supervision for generalization under distribution shifts,

    Y . Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt, “Test- time training with self-supervision for generalization under distribution shifts,” inProceedings of the International Conference on Machine Learning (ICML), 2020, pp. 9229–9248

    2020

  4. [13]

    Ttt++: When does self-supervised test-time training fail or thrive?

    Y . Liu, P. Kothari, B. Van Delft, B. Bellot-Gurlet, T. Mordan, and A. Alahi, “Ttt++: When does self-supervised test-time training fail or thrive?” inAdvances in Neural Information Processing Systems (NeurIPS), 2021, pp. 21 808–21 820

    2021

  5. [14]

    Video test-time adaptation for action recognition,

    W. Lin, M. J. Mirza, M. Kozinski, H. Possegger, H. Kuehne, and H. Bischof, “Video test-time adaptation for action recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 22 952–22 961

    2023

  6. [15]

    Exploring motion cues for video test-time adaptation,

    R. Zeng, Q. Deng, H. Xu, S. Niu, and J. Chen, “Exploring motion cues for video test-time adaptation,” inProceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 1840–1850

    2023

  7. [16]

    Temporal coherent test time optimization for robust video classification,

    C. Yi, S. Yang, Y . Wang, H. Li, Y . Tan, and A. C. Kot, “Temporal coherent test time optimization for robust video classification,” inPro- ceedings of the International Conference on Learning Representations (ICLR), 2023

    2023

  8. [17]

    Ast: Audio spectrogram trans- former,

    Y . Gong, Y .-A. Chung, and J. Glass, “Ast: Audio spectrogram trans- former,” inAnnual Conference of the International Speech Communica- tion Association (INTERSPEECH), 2021, pp. 571–575

    2021

  9. [18]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inProceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019, pp. 4171–4186

    2019

  10. [19]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,” inAdvances in Neural Information Processing Systems (NeurIPS), 2020, pp. 1877–1901

    2020

  11. [20]

    Benchmarking micro-action recognition: Dataset, methods, and applications,

    D. Guo, K. Li, B. Hu, Y . Zhang, and M. Wang, “Benchmarking micro-action recognition: Dataset, methods, and applications,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 7, pp. 6238–6252, 2024

    2024

  12. [21]

    Repetitive action counting with hybrid temporal relation modeling,

    K. Li, X. Peng, D. Guo, X. Yang, and M. Wang, “Repetitive action counting with hybrid temporal relation modeling,”IEEE Transactions on Multimedia, 2025

    2025

  13. [22]

    Prototypical calibrating ambiguous samples for micro-action recogni- tion,

    K. Li, D. Guo, G. Chen, C. Fan, J. Xu, Z. Wu, H. Fan, and M. Wang, “Prototypical calibrating ambiguous samples for micro-action recogni- tion,”arXiv preprint arXiv:2412.14719, 2024

    2024 arXiv

  14. [23]

    Test-time classifier adjustment module for model-agnostic domain generalization,

    Y . Iwasawa and Y . Matsuo, “Test-time classifier adjustment module for model-agnostic domain generalization,” inAdvances in Neural Information Processing Systems (NeurIPS), 2021, pp. 2427–2440

    2021

  15. [24]

    Revisiting realistic test-time training: Se- quential inference and adaptation by anchored clustering,

    Y . Su, X. Xu, and K. Jia, “Revisiting realistic test-time training: Se- quential inference and adaptation by anchored clustering,” inAdvances in Neural Information Processing Systems (NeurIPS), 2022, pp. 17 543– 17 555

    2022

  16. [26]

    Sita: Single image test-time adaptation,

    A. Khurana, S. Paul, P. Rai, S. Biswas, and G. Aggarwal, “Sita: Single image test-time adaptation,”arXiv preprint arXiv:2112.02355, 2021

    2021 arXiv

  17. [27]

    Parameter- free online test-time adaptation,

    M. Boudiaf, R. Mueller, I. Ben Ayed, and L. Bertinetto, “Parameter- free online test-time adaptation,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 8344– 8353

    2022

  18. [28]

    Camera-aware recurrent learn- ing and earth mover’s test-time adaption for generalizable person re- identification,

    K. Chen, T. Gong, and L. Zhang, “Camera-aware recurrent learn- ing and earth mover’s test-time adaption for generalizable person re- identification,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 1, pp. 357–370, 2023

    2023

  19. [29]

    Question type-aware debiasing for test-time visual question answering model adaptation,

    J. Liu, J. Xie, F. Zhou, and S. He, “Question type-aware debiasing for test-time visual question answering model adaptation,”IEEE Transac- tions on Circuits and Systems for Video Technology, 2024

    2024

  20. [30]

    Ttagaze: Self- supervised test-time adaptation for personalized gaze estimation,

    Y . Wu, G. Chen, L. Ye, Y . Jia, Z. Liu, and Y . Wang, “Ttagaze: Self- supervised test-time adaptation for personalized gaze estimation,”IEEE Transactions on Circuits and Systems for Video Technology, 2024

    2024

  21. [31]

    What makes training multi-modal classification networks hard?

    W. Wang, D. Tran, and M. Feiszli, “What makes training multi-modal classification networks hard?” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 12 695– 12 705

    2020

  22. [32]

    Self-supervised learning by cross-modal audio-video cluster- ing,

    H. Alwassel, D. Mahajan, B. Korbar, L. Torresani, B. Ghanem, and D. Tran, “Self-supervised learning by cross-modal audio-video cluster- ing,” inAdvances in Neural Information Processing Systems (NeurIPS), 2020, pp. 9758–9770

    2020

  23. [33]

    Look, listen and learn,

    R. Arandjelovic and A. Zisserman, “Look, listen and learn,” inProceed- ings of the IEEE international conference on computer vision, 2017, pp. 609–617

    2017

  24. [34]

    Epic-fusion: Audio-visual temporal binding for egocentric action recognition,

    E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen, “Epic-fusion: Audio-visual temporal binding for egocentric action recognition,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019, pp. 5492–5501

    2019

  25. [35]

    Exploring multimodal video representation for action recognition,

    C. Wang, H. Yang, and C. Meinel, “Exploring multimodal video representation for action recognition,” inProceedings of International Joint Conference on Neural Networks (IJCNN), 2016

    2016

  26. [36]

    Slowfast networks for video recognition,

    C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), 2019, pp. 6202–6211

    2019

  27. [37]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems (NeurIPS), 2017, pp. 5998– 6008

    2017

  28. [38]

    Polyvit: Co-training vision transformers on images, videos and audio,

    V . Likhosherstov, A. Arnab, K. Choromanski, M. Lucic, Y . Tay, and M. Dehghani, “Polyvit: Co-training vision transformers on images, videos and audio,”Transactions on Machine Learning Research, vol. 2023, 2023

    2023

  29. [39]

    Attention bottlenecks for multimodal fusion,

    A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, “Attention bottlenecks for multimodal fusion,” inAdvances in Neural Information Processing Systems (NeurIPS), 2021, pp. 14 200–14 213

    2021

  30. [40]

    Mm-vit: Multi-modal video transformer for compressed video action recognition,

    J. Chen and C. M. Ho, “Mm-vit: Multi-modal video transformer for compressed video action recognition,” inProceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2022, pp. 1910–1921

    2022

  31. [41]

    Multimodal video summa- rization via time-aware transformers,

    X. Shang, Z. Yuan, A. Wang, and C. Wang, “Multimodal video summa- rization via time-aware transformers,” inProceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 1756–1765

    2021

  32. [42]

    Bootstrapping audio-visual video segmentation by strengthening audio cues,

    T. Chen, Z. Tan, T. Gong, Q. Chu, Y . Wu, B. Liu, N. Yu, L. Lu, and J. Ye, “Bootstrapping audio-visual video segmentation by strengthening audio cues,”IEEE Transactions on Circuits and Systems for Video Technology, 2024

    2024

  33. [43]

    Multimodal imbalance-aware gradient modulation for weakly-supervised audio-visual video parsing,

    J. Fu, J. Gao, B.-K. Bao, and C. Xu, “Multimodal imbalance-aware gradient modulation for weakly-supervised audio-visual video parsing,” IEEE Transactions on Circuits and Systems for Video Technology, 2023. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 14

    2023

  34. [44]

    Cross-domain first person audio-visual action recognition through relative norm align- ment,

    M. Planamente, C. Plizzari, E. Alberti, and B. Caputo, “Cross-domain first person audio-visual action recognition through relative norm align- ment,”arXiv preprint arXiv:2106.01689, 2021

    2021 arXiv

  35. [45]

    Domain generalization through audio-visual relative norm align- ment in first person action recognition,

    ——, “Domain generalization through audio-visual relative norm align- ment in first person action recognition,” inProceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2022, pp. 1807–1818

    2022

  36. [46]

    Epic-kitchens-100 unsu- pervised domain adaptation challenge for action recognition 2021: Team m3em technical report,

    L. Yang, Y . Huang, Y . Sugano, and Y . Sato, “Epic-kitchens-100 unsu- pervised domain adaptation challenge for action recognition 2021: Team m3em technical report,”arXiv preprint arXiv:2106.10026, 2021

    2021 arXiv

  37. [47]

    Audio-adaptive activity recognition across video domains,

    Y . Zhang, H. Doughty, L. Shao, and C. G. Snoek, “Audio-adaptive activity recognition across video domains,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 13 791–13 800

    2022

  38. [48]

    Evaluating prediction-time batch normalization for robustness under covariate shift,

    Z. Nado, S. Padhy, D. Sculley, A. D’Amour, B. Lakshminarayanan, and J. Snoek, “Evaluating prediction-time batch normalization for robustness under covariate shift,”arXiv preprint arXiv:2006.10963, 2020

    2006 arXiv

  39. [49]

    Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation,

    J. Liang, D. Hu, and J. Feng, “Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation,” inProceedings of the International Conference on Machine Learning (ICML), 2020, pp. 6028–6039

    2020

  40. [50]

    Improving robustness against common corruptions by co- variate shift adaptation,

    S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge, “Improving robustness against common corruptions by co- variate shift adaptation,” inAdvances in Neural Information Processing Systems (NeurIPS), 2020, pp. 11 539–11 551

    2020

  41. [51]

    Entropy is not enough for test-time adaptation: From the perspective of disentangled factors,

    J. Lee, D. Jung, S. Lee, J. Park, J. Shin, U. Hwang, and S. Yoon, “Entropy is not enough for test-time adaptation: From the perspective of disentangled factors,” inThe Twelfth International Conference on Learning Representations, ICLR, 2024

    2024

  42. [52]

    Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction,

    R. A. Marsden, M. D ¨obler, and B. Yang, “Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 2555–2565

    2024

  43. [53]

    Audiovisual moments in time: A large-scale annotated dataset of audiovisual actions,

    M. Joannou, P. Rotshtein, and U. Noppeney, “Audiovisual moments in time: A large-scale annotated dataset of audiovisual actions,”arXiv preprint arXiv:2308.09685, 2023

    2023 arXiv

  44. [54]

    Audio-visual event localization in unconstrained videos,

    Y . Tian, J. Shi, B. Li, Z. Duan, and C. Xu, “Audio-visual event localization in unconstrained videos,” inProceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 247–263

    2018

  45. [55]

    Audio set: An ontology and human- labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human- labeled dataset for audio events,” inProceedings of the IEEE Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 776– 780

    2017

  46. [56]

    Ucf101: A dataset of 101 human actions classes from videos in the wild,

    K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,”arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  47. [57]

    The kinetics human action video dataset,

    W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijaya- narasimhan, F. Viola, T. Green, T. Back, P. Natsevet al., “The kinetics human action video dataset,”arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  48. [58]

    Large-scale robustness analysis of video action recognition models,

    M. C. Schiappa, N. Biyani, S. Vyas, H. Palangi, V . Vineet, and Y . Rawat, “Large-scale robustness analysis of video action recognition models,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 14 698–14 708

    2023

  49. [59]

    Benchmarking the ro- bustness of spatial-temporal models against corruptions,

    C. Yi, S. Yang, H. Li, Y .-p. Tan, and A. Kot, “Benchmarking the ro- bustness of spatial-temporal models against corruptions,” inProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021

    2021

  50. [60]

    Tam: Temporal adaptive module for video recognition,

    Z. Liu, L. Wang, W. Wu, C. Qian, and T. Lu, “Tam: Temporal adaptive module for video recognition,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), 2021, pp. 13 708–13 718

    2021

  51. [61]

    Tsm: Temporal shift module for efficient video understanding,

    J. Lin, C. Gan, and S. Han, “Tsm: Temporal shift module for efficient video understanding,” inProceedings of the IEEE International Confer- ence on Computer Vision (ICCV), 2020, pp. 7083–7093

    2020

  52. [62]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778

    2016

  53. [63]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  54. [64]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic, “The claude 3 model family: Opus, sonnet, haiku.” [Online]. Available: https://api.semanticscholar.org/CorpusID:268232499

  55. [65]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  56. [66]

    Binggpt: Desktop application of new bing’s ai-powered chat,

    dice2o, “Binggpt: Desktop application of new bing’s ai-powered chat,” GitHub Repository, 2024. [Online]. Available: https://github.com/ dice2o/BingGPT

    2024

  57. [67]

    Audio mamba: Bidirectional state space model for audio representation learning,

    M. H. Erol, A. Senocak, J. Feng, and J. S. Chung, “Audio mamba: Bidirectional state space model for audio representation learning,”IEEE Signal Processing Letters, 2024

    2024

  58. [68]

    Beats: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, “Beats: Audio pre-training with acoustic tokenizers,” inInternational Conference on Machine Learning, ICML, 2023, pp. 5178–5193

    2023

  59. [69]

    Tim: A time interval machine for audio-visual action recognition,

    J. Chalk, J. Huh, E. Kazakos, A. Zisserman, and D. Damen, “Tim: A time interval machine for audio-visual action recognition,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 153–18 163

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.