Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

MUG: Pseudo Labeling Augmented Audio-Visual Mamba Network for Audio-Visual Video Parsing

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MUG, a Mamba-Transformer network trained with pseudo-label cross-modal random combination, claims new best scores on every segment- and event-level metric of the LLP audio-visual parsing benchmark.

desk verdict Plausible gains and a reusable augmentation idea, but the headline SOTA numbers are undercut by test-set batch selection and an unquantified manual annotation step. read the letter →

arxiv 2507.01384 v2 pith:C775RCOU submitted 2025-07-02 cs.CV

Langyu Wang , Bingke Zhu , Yingying Chen , Yiyuan Zhang , Ming Tang , Jinqiao Wang This is my paper
classification cs.CV
keywords audio-visualvideoparsingweaklysupervisedlearningpseudo-labelaugmentationcross-modalrandomcombinationMambastatespacemodelmultimodalfusionLLPdatasettemporaleventlocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that weakly-supervised audio-visual video parsing can be improved on both per-segment and whole-event accuracy at once by fixing the pseudo-label signal and reworking the fusion architecture. It introduces MUG, which takes segment-level pseudo-labels from an earlier method, manually corrects the visual pseudo-labels that are empty, and then generates new training videos by splicing the visual track of one video with the audio track of another. A Mamba-Transformer network processes the augmented data, sharing state-transition parameters across modalities while adding a dynamic scanning branch, a feature enhancement module, and text-feature guidance from CLIP/CLAP pseudo-label semantics. On the LLP benchmark, MUG reports improved scores on every segment- and event-level metric relative to prior methods, including 2.1 points on visual segment F1 (66.6 vs. 64.4) and 1.2 points on audio segment F1 (65.4 vs. 64.2). The central claim is that cleaner pseudo-labels, cross-modal data augmentation, and selective state-space fusion together let a weakly supervised parser resolve fine-grained segments and longer events simultaneously.

What carries the argument

The load-bearing components are Cross-Modal Random Combination (CMRC), a data augmentation scheme that splices the visual track of one video with the audio track of another and labels the result with the union of their unimodal pseudo-labels, and the AV-Mamba network, a Mamba-Transformer hybrid. Mamba is a selective state-space sequence model that scales linearly with sequence length and models causal order; the paper uses it in a Mamba-based attention module, an adaptive fusion module that shares the state-transition matrix between modalities while keeping other SSM parameters independent, a dynamic scanning branch, and a feature enhancement module. Pseudo-label semantic interaction rounds out the machinery by encoding event names with frozen CLIP and CLAP text encoders and fusing the resulting semantic vectors into the visual and audio features as a denoising constraint.

What would settle it

Retrain MUG with the original unannotated VALOR pseudo-labels, keeping CMRC unchanged; if visual segment-level F1 on the LLP test set stays at 66.5, manual annotation is not load-bearing, and if it falls back toward the 65.2 of the no-CMRC ablation, the reported gain is carried by hand-added supervision.

Watch

Extended reading notes

Core claim

On the LLP dataset, MUG establishes a new best result on every reported metric by attacking two bottlenecks. The data bottleneck is empty visual pseudo-labels: the prior VALOR thresholding leaves many visual segments with no predicted event, and the authors manually annotate those obviously wrong labels, discarding any video that cannot be annotated into the 25 predefined categories. The architecture bottleneck is that transformers struggle with roughly 2000 visual tokens while pure Mamba models lose single-frame spatial modeling; the paper's answer is a Mamba-Transformer hybrid with shared SSM state-transition matrices between audio and visual streams, a dynamic scan branch to avoid forgetting early tokens, a Mamba feature enhancement module that amplifies cross-modal similarities, and a pseudo-label semantic interaction module that injects CLIP/CLAP text features as a denoising prior. The paper asserts that both fixes are needed: removing CMRC drops visual segment F1 from 66.5 to 65.2, and removing the Mamba feature enhancement module drops it to 64.6, with similar degradations across all metrics.

Load-bearing premise

The load-bearing premise is that the manual annotation of empty visual pseudo-labels is accurate and unbiased enough to provide clean segment supervision, yet the paper reports neither how many labels were corrected nor an ablation that isolates this step.

Editorial extensions

If this is right

  • If the reported gains hold, a weakly supervised parsing system can beat the previous best on segment-level and event-level metrics at the same time, rather than trading one off against the other.
  • CMRC is a data-side module and can be dropped onto other audio-visual parsing baselines; the paper shows it raises the scores of HAN, MGN, and JoMoLD.
  • The Mamba-Transformer design achieves these scores with 7.6M parameters, outperforming both a 6.5M CNN baseline and a 19.3M Transformer baseline, so the gain is not simply a larger model.
  • Using text features derived from pseudo-label captions as a unimodal denoising prior is presented as a reusable way to keep one modality from contaminating the other.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the manual annotation step corrects a large number of empty visual pseudo-labels, the effective supervision is partly human segment labels rather than pure weak video-level labels; counting the corrected labels and ablating manual annotation alone would settle how much of the gain is human-added.
  • CMRC-style cross-modal splicing should transfer to other weakly supervised audio-visual tasks where unimodal pseudo-labels can be extracted, but only when one modality shows a similar 'empty label' pathology.
  • Sharing the SSM state-transition matrix between modalities embodies a testable assumption that audio and visual events evolve under similar hidden-state dynamics; an ablation with fully independent SSM parameters would show whether sharing itself, rather than the scan structure, drives the fusion gain.
  • The text-semantic fusion could be pushed further by generating captions that include temporal context, potentially reducing or removing the need for manual pseudo-label correction in future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes MUG, a weakly-supervised audio-visual video parsing (AVVP) method that combines a pseudo-label augmentation strategy (CMRC) with a Mamba-Transformer architecture. The authors extract pseudo-labels from VALOR, manually correct empty visual pseudo-labels (MA), and generate new training videos by randomly combining visual and audio tracks from different videos. They also introduce an audio-visual Mamba network with adaptive fusion, a feature enhancement module, and a pseudo-label semantic interaction module using CLIP/CLAP text features. Experiments on the LLP dataset report state-of-the-art F1 scores on all metrics, e.g., +2.1% visual segment-level F1 over CoLeaF. Code is released.

Significance. If the reported results are validated, the CMRC augmentation is a simple and transferable idea that can be applied to any AVVP baseline, as demonstrated in Table 4, and the Mamba-Transformer architecture offers a plausible efficiency/accuracy trade-off. However, the current experimental protocol contains two load-bearing weaknesses: the augmentation batch size is selected using test-set performance (Table 3), and the manual annotation contribution is not separately ablated. These issues must be addressed before the SOTA claim can be accepted. On the positive side, the paper provides code, applies CMRC to multiple baselines, and includes qualitative comparisons.

major comments (5)
  1. [Introduction vs. §3.2] The label of a CMRC-generated video is defined as the 'intersection' of the visual and audio pseudo-labels in the Introduction, but as the 'union' in Section 3.2. This is not a terminological nuance: the augmentation's training signal depends on which operation is used, and the consistency of the pseudo-labels with the LLP weak labels differs accordingly. Please correct the inconsistency and specify which definition was used for all reported experiments.
  2. [§3.2, Table 3] The augmentation batch size is selected by evaluating five batches on the test set and adopting Batch4 for the final model. No validation set is used in this selection, despite the LLP dataset having a validation split. Because the final numbers in Table 1 are the best of five test-set evaluations, the reported improvements (e.g., +2.1% visual segment-level F1) are optimistically biased and the claim of a SOTA improvement is not established. Please select hyperparameters on the validation set and report the corresponding test performance, or provide a correction for multiple comparisons.
  3. [§3.2, Table 2] The manual annotation (MA) step is not quantified or ablated. The paper does not report how many pseudo-labels were manually corrected, what criteria were used, or who performed the annotation. The wo/CMRC ablation removes both MA and CMRC together, so the gain attributed to data augmentation may actually come from the hand-added labels. Please add an ablation that uses MA alone, and report the number and distribution of corrected labels.
  4. [Table 3] The choice of Batch4 is not fully justified by the numbers: Batch5 achieves higher segment-level audio F1 (66.3 vs. 65.4) and higher Event@AV (64.9 vs. 64.7) than Batch4, while Batch4 is better on other metrics. Without a predefined selection criterion or validation-based metric, the selection is arbitrary. Please define the criterion or use a validation set.
  5. [§3.3, Eq. (3)] Equation (3) uses $f^a_t$ in the second factor even when $m=v$, and the notation $S^m_t(W^m_t(f^m_t)\otimes f^m_t)$ is ambiguous. Please clarify the intended operation and ensure the equation is dimensionally consistent.
minor comments (6)
  1. [Abstract] The abstract contains a typo: 'a audio-visual' should be 'an audio-visual', and 'e.g,,' has a double comma.
  2. [§3.3] Section 3.3 refers to 'recent work [46]' for segment-wise pseudo-labels; reference [46] is CutMix, not VALOR [23]. Please fix the citation.
  3. [Table 4] In Table 4, 'CMRC+JoMoLD62.3' is missing a space and the alignment is off; please format the table consistently.
  4. [§3.3, Eq. (2)] Equation (2) uses a semicolon inside the argument; clarify whether this denotes concatenation.
  5. [Figure 2] Figure 2 is referenced but its caption does not explain the intersection/union operation; please make the figure self-contained.
  6. [Abstract] The phrase 'we annotate some of the pseudo-labels based on previous work' is vague; specify in the abstract that only empty visual pseudo-labels are manually corrected.

Circularity Check

1 steps flagged · score 4.0 of 10

Augmentation batch size is selected on the test set in Section 3.2, so the headline gains are partly the result of test-set selection; the core method is otherwise self-contained.

  1. fitted input called prediction [Section 3.2 (Data augmentation in AVVP); Section 4.3 (Table 3); Section 4.2 (Table 1)]
    "We separately test data augmentation for different batches to find the critical point of this method. [...] The results indicate that training the model with the combined data can effectively enhance accuracy (batch4)."

    Table3 reports test-set F-scores for five augmentation-batch sizes, and the text selects 'batch4' as the effective configuration before the Table1 comparison. Since the identical test set is used both to choose the batch size and to declare the SOTA improvement, the reported gains over CoLeaF are the argmax of five test-set evaluations rather than the outcome of a pre-specified method. The paper does not hold out the validation set for this choice or correct for multiple comparisons, so the central 'improvement' is statistically forced in part by the selection procedure, not derived independently of the test labels.

full rationale

The evaluation is on the external LLP benchmark and compares against external baselines, so the core architectural derivation is not circular: the Mamba-based attention, adaptive fusion, feature enhancement, and pseudo-label semantic interaction are defined by their own equations and trained with standard weak supervision. There is a self-citation to the authors' own LINK paper [35], but it is used only to motivate text cues in PLSIM and is not load-bearing. The manual annotation step in Section 3.2 injects external human knowledge rather than deriving the result from the benchmark, though its quantity and criteria are unstated. The one substantive circularity-burden signal is the selection of the CMRC batch size: Section 3.2 says the batches were tested separately to find the critical point, Table3 reports test-set metrics for all five batches, and the final Table1 configuration is the batch that scored best on that same test set. This means the headline SOTA gains are partially manufactured by test-set selection, not obtained from a fixed method. Because the architecture and augmentation still contribute independently and the reported result is not mathematically forced by a definitional identity, the score is moderate rather than severe.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the accuracy of VALOR pseudo-labels, an unquantified manual annotation step, and a test-set-selected augmentation scale; these are the main external inputs the reader must accept without independent verification.

free parameters (2)
  • CMRC augmentation batch scale (Batch4) = 12096 generated samples (1x)
    Selected as the final augmentation size based on F1 results across five batch sizes (Table3); this is hyperparameter selection on the benchmark itself, inflating the reported gains.
  • Manual annotation effort = Not quantified
    The paper manually annotates empty visual pseudo-labels but does not report the number of annotated segments, annotator instructions, or agreement; this hidden human supervision is a free input affecting results.
assumptions (4)
  • domain assumption VALOR pseudo-labels [23] are accurate enough after manual correction to serve as segment-level supervision.
    The entire CMRC augmentation and PLSIM rely on these pseudo-labels; no independent verification is provided beyond the manual correction step.
  • domain assumption Combining the visual track of one video with the audio track of another produces valid training samples whose label is the union of the two pseudo-label sets.
    Assumes segment-level independence and that the synthetic audio-visual pair is representative of real-world modality mismatch; stated in Section 3.2.
  • domain assumption Mamba-style causal sequence modeling is suitable for segment-level audio and visual features in AVVP.
    The paper argues this empirically but provides no theoretical justification; it is a design choice.
  • domain assumption The LLP weak video-level labels are correct and the 25 event categories cover all relevant events.
    The authors themselves note that the 25 categories cannot encompass all events, yet the manual annotation leaves non-category events unannotated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MUG: Pseudo Labeling Augmented Audio-Visual Mamba Network for Audio-Visual Video Parsing." pith.science (2026). https://pith.science/paper/C775RCOU

@misc{pith2026250701384,
  author       = {Pith},
  title        = {Pith review of: MUG: Pseudo Labeling Augmented Audio-Visual Mamba Network for Audio-Visual Video Parsing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C775RCOU}},
  note         = {Machine review of arXiv:2507.01384}
}
read the original abstract

The weakly-supervised audio-visual video parsing (AVVP) aims to predict all modality-specific events and locate their temporal boundaries. Despite significant progress, due to the limitations of the weakly-supervised and the deficiencies of the model architecture, existing methods are lacking in simultaneously improving both the segment-level prediction and the event-level prediction. In this work, we propose a audio-visual Mamba network with pseudo labeling aUGmentation (MUG) for emphasising the uniqueness of each segment and excluding the noise interference from the alternate modalities. Specifically, we annotate some of the pseudo-labels based on previous work. Using unimodal pseudo-labels, we perform cross-modal random combinations to generate new data, which can enhance the model's ability to parse various segment-level event combinations. For feature processing and interaction, we employ a audio-visual mamba network. The AV-Mamba enhances the ability to perceive different segments and excludes additional modal noise while sharing similar modal information. Our extensive experiments demonstrate that MUG improves state-of-the-art results on LLP dataset in all metrics (e.g,, gains of 2.1% and 1.2% in terms of visual Segment-level and audio Segment-level metrics). Our code is available at https://github.com/WangLY136/MUG.

Figures

Figures reproduced from arXiv: 2507.01384 by the authors.

Figure 1
Figure 1. Fig.1 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Fig.2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Fig.3 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Fig.4 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniSkip-Mamba: A Frequency-Aware State Space Model for Audio-Visual Temporal Forgery Localization

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Skip-scanning Mamba with unified audio-visual sequences reaches 63.4% AP@0.95 on LAV-DF and 63.58% mAP on AV-Deepfake1M by regularizing toward low/mid-frequency forgery cues.

Reference graph

Works this paper leans on

50 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [46]

    Cutmix: Regu- larization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 6023–6032, 2019. 3

    work page 2019

  2. [23]

    Modality-independent teachers meet weakly-supervised audio-visual event parser

    Yung-Hsuan Lai, Yen-Chun Chen, and Frank Wang. Modality-independent teachers meet weakly-supervised audio-visual event parser. Advances in Neural Information Processing systems, 36:73633–73651, 2023. 2, 3, 4, 5, 7

    work page 2023

  3. [1]

    Yolov4: Optimal speed and accuracy of object detection

    Alexey Bochkovskiy, Chien-Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934, 2020. 3

    arXiv 2004

  4. [2]

    Cm-pie: Cross-modal perception for interactive-enhanced audio-visual video pars- ing

    Yaru Chen, Ruohao Guo, Xubo Liu, Peipei Wu, Guangyao Li, Zhenbo Li, and Wenwu Wang. Cm-pie: Cross-modal perception for interactive-enhanced audio-visual video pars- ing. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8421–8425. IEEE, 2024. 7

    work page 2024

  5. [3]

    Joint-modal label denois- ing for weakly-supervised audio-visual video parsing

    Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, and Limin Wang. Joint-modal label denois- ing for weakly-supervised audio-visual video parsing. In European Conference on Computer Vision, pages 431–448. Springer, 2022. 3, 6, 7

    work page 2022

  6. [4]

    Autoaugment: Learning augmentation strategies from data

    Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasude- van, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 113–123, 2019. 3

    work page 2019

  7. [5]

    Randaugment: Practical automated data augmen- tation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmen- tation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020. 3

    2020

  8. [6]

    Transformers are ssms: General- ized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: General- ized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. 2

    arXiv 2024

Show all 50 references
  1. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3

    2010 arXiv

  2. [8]

    Cross-modal prompts: Adapting large pre- trained models for audio-visual downstream tasks

    Haoyi Duan, Yan Xia, Zhou Mingze, Li Tang, Jieming Zhu, and Zhou Zhao. Cross-modal prompts: Adapting large pre- trained models for audio-visual downstream tasks. Advances in Neural Information Processing Systems, 36, 2024. 2, 7

    2024

  3. [9]

    Revisit weakly-supervised audio-visual video parsing from the lan- guage perspective

    Yingying Fan, Yu Wu, Bo Du, and Yutian Lin. Revisit weakly-supervised audio-visual video parsing from the lan- guage perspective. Advances in Neural Information Process- ing Systems, 36, 2024. 2, 3

    2024

  4. [10]

    Col- lecting cross-modal presence-absence evidence for weakly- supervised audio-visual event perception

    Junyu Gao, Mengyuan Chen, and Changsheng Xu. Col- lecting cross-modal presence-absence evidence for weakly- supervised audio-visual event perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18827–18836, 2023. 2, 7

    2023

  5. [11]

    Avs-mamba: Ex- ploring temporal and multi-modal mamba for audio-visual segmentation

    Sitong Gong, Yunzhi Zhuge, Lu Zhang, Yifan Wang, Ping- ping Zhang, Lijun Wang, and Huchuan Lu. Avs-mamba: Ex- ploring temporal and multi-modal mamba for audio-visual segmentation. arXiv preprint arXiv:2501.07810, 2025. 2

    2025 arXiv

  6. [12]

    Explaining and harnessing adversarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. 3

    2014 arXiv

  7. [13]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 2, 3

    2023 arXiv

  8. [14]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R ´e. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. 3

    2021 arXiv

  9. [15]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R ´e. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing sys- tems, 34:572–585, 2021. 3

    2021

  10. [16]

    Liquid structural state-space models

    Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. arXiv preprint arXiv:2209.12951, 2022. 3

    2022 arXiv

  11. [17]

    Mambavision: A hy- brid mamba-transformer vision backbone

    Ali Hatamizadeh and Jan Kautz. Mambavision: A hy- brid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083, 2024. 2

    2024 arXiv

  12. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6

    2016

  13. [19]

    Augmix: A simple data processing method to improve robustness and uncertainty

    Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019. 3

    1912 arXiv

  14. [20]

    Cnn archi- tectures for large-scale audio classification

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn archi- tectures for large-scale audio classification. In 2017 ieee in- ternational conference on acoustics, spe...

    2017

  15. [21]

    Localmamba: Visual state space model with windowed selective scan

    Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. arXiv preprint arXiv:2403.09338,

    arXiv

  16. [22]

    Learning tem- porally invariant and localizable features via data augmenta- tion for video recognition

    Taeoh Kim, Hyeongmin Lee, MyeongAh Cho, Ho Seong Lee, Dong Heon Cho, and Sangyoun Lee. Learning tem- porally invariant and localizable features via data augmenta- tion for video recognition. In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceed- ings...

    2020

  17. [24]

    Coupled mamba: Enhanced multi-modal fu- sion with coupled state space model

    Wenbing Li, Hang Zhou, Junqing Yu, Zikai Song, and Wei Yang. Coupled mamba: Enhanced multi-modal fu- sion with coupled state space model. arXiv preprint arXiv:2405.18014, 2024. 3

    2024 arXiv

  18. [25]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024. 2

    2024 arXiv

  19. [26]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. 3

    2024

  20. [27]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 3

    2021

  21. [28]

    Rgbt tracking via all-layer multimodal inter- actions with progressive fusion mamba

    Andong Lu, Wanyu Wang, Chenglong Li, Jin Tang, and Bin Luo. Rgbt tracking via all-layer multimodal inter- actions with progressive fusion mamba. arXiv preprint arXiv:2408.08827, 2024. 5

    2024 arXiv

  22. [29]

    Multi-modal grouping network for weakly-supervised audio-visual video parsing

    Shentong Mo and Yapeng Tian. Multi-modal grouping network for weakly-supervised audio-visual video parsing. Advances in Neural Information Processing Systems , 35: 34722–34733, 2022. 2, 7

    2022

  23. [30]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  24. [31]

    Coleaf: A contrastive-collaborative learning framework for weakly supervised audio-visual video pars- ing

    Faegheh Sardari, Armin Mustafa, Philip JB Jackson, and Adrian Hilton. Coleaf: A contrastive-collaborative learning framework for weakly supervised audio-visual video pars- ing. In European Conference on Computer Vision, pages 1–

  25. [32]

    Simplified state space layers for sequence modeling

    Jimmy TH Smith, Andrew Warrington, and Scott W Linder- man. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933, 2022. 3

    2022 arXiv

  26. [33]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chen- liang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on com- puter vision (ECCV), pages 247–263, 2018. 1

    2018

  27. [34]

    Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing

    Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceed- ings, Part III 16 , pages 436–454. Springer, 2020. 1, 2, 6, 7

    2020

  28. [35]

    Link: Adaptive modality interaction for audio-visual video parsing

    Langyu Wang, Bingke Zhu, Yingying Chen, and Jinqiao Wang. Link: Adaptive modality interaction for audio-visual video parsing. pages 1–5, 2025. 2, 5

    2025

  29. [36]

    Cbam: Convolutional block attention module

    Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018. 4

    2018

  30. [37]

    Exploring heterogeneous clues for weakly-supervised audio-visual video parsing

    Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1326–1335, 2021. 2

    2021

  31. [38]

    Dual attention matching for audio-visual event localization

    Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 6292–6300, 2019. 1

    2019

  32. [39]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Si...

    2023

  33. [40]

    Fusionmamba: Dy- namic feature enhancement for multimodal image fusion with mamba

    Xinyu Xie, Yawen Cui, Chio-In Ieong, Tao Tan, Xiaozhi Zhang, Xubin Zheng, and Zitong Yu. Fusionmamba: Dy- namic feature enhancement for multimodal image fusion with mamba. arXiv preprint arXiv:2404.09498, 2024. 5

    2024 arXiv

  34. [41]

    Language- driven all-in-one adverse weather removal

    Hao Yang, Liyuan Pan, Yan Yang, and Wei Liang. Language- driven all-in-one adverse weather removal. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24902–24912, 2024. 5

    2024

  35. [42]

    Depmamba: Progressive fusion mamba for multimodal depression detec- tion

    Jiaxin Ye, Junping Zhang, and Hongming Shan. Depmamba: Progressive fusion mamba for multimodal depression detec- tion. arXiv preprint arXiv:2409.15936, 2024. 5, 7

    2024 arXiv

  36. [43]

    Text-if: Leveraging semantic text guidance for degradation-aware and interactive image fusion

    Xunpeng Yi, Han Xu, Hao Zhang, Linfeng Tang, and Ji- ayi Ma. Text-if: Leveraging semantic text guidance for degradation-aware and interactive image fusion. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27026–27035, 2024. 5

    2024

  37. [44]

    Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video pars- ing

    Jiashuo Yu, Ying Cheng, Rui-Wei Zhao, Rui Feng, and Yue- jie Zhang. Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video pars- ing. In Proceedings of the 30th ACM international confer- ence on multimedia, pages 6241–6249, 2022. 2, 7

    2022

  38. [45]

    Mambaout: Do we really need mamba for vision? arXiv preprint arXiv:2405.07992,

    Weihao Yu and Xinchao Wang. Mambaout: Do we really need mamba for vision? arXiv preprint arXiv:2405.07992,

    arXiv

  39. [47]

    Videomix: Rethinking data augmentation for video classification

    Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongy- oon Han, and Jinhyung Kim. Videomix: Rethinking data augmentation for video classification. arXiv preprint arXiv:2012.03457, 2020. 3

    2012 arXiv

  40. [48]

    Don’t judge by the look: A motion coherent augmenta- tion for video recognition

    Yitian Zhang, Yue Bai, Huan Wang, Yizhou Wang, and Yun Fu. Don’t judge by the look: A motion coherent augmenta- tion for video recognition. arXiv preprint arXiv:2403.09506,

    arXiv

  41. [49]

    Label-anticipated event dis- entanglement for audio-visual video parsing

    Jinxing Zhou, Dan Guo, Yuxin Mao, Yiran Zhong, Xiao- jun Chang, and Meng Wang. Label-anticipated event dis- entanglement for audio-visual video parsing. arXiv preprint arXiv:2407.08126, 2024. 3, 5, 7

    2024 arXiv

  42. [50]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 3, 7

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.