Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

The paper claims that anchoring audio queries to a prototype memory bank and coupling them with delayed bidirectional cross-attention yields state-of-the-art audio-visual segmentation, with J&F scores of 92.4, 75.1, and 52.6 on single-sourc

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:27 UTC pith:XEXJPL6U

load-bearing objection DDAVS is a solid, incremental SOTA paper on AVS with a real but under-validated contrastive mechanism; worth refereeing, but the authors need to release code, λs, and run significance tests. the 3 major comments →

arxiv 2512.20117 v2 pith:XEXJPL6U submitted 2025-12-23 cs.CV cs.SDeess.AS

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Jingqi Tian , Yiheng Du , Haoji Zhang , Yuji Wang , Isaac Ning Lee , Xulong Bai , Tianrui Zhu , Jingxuan Niu
show 1 more author
Yansong Tang
This is my paper
classification cs.CV cs.SDeess.AS
keywords audio-visual segmentationmulti-source disentanglementprototype memory bankcontrastive learningdelayed bidirectional alignmentaudio querycross-modal alignmentmulti-instance segmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper aims to fix two persistent failures in audio-visual segmentation: when multiple objects make sound at once, models tend to favor louder or larger objects, and when sound and vision are misaligned (e.g., off-screen audio), models mislocalize. The authors propose DDAVS, which first turns audio into a small set of disentangled semantic queries anchored to a fixed prototype memory bank built from clean single-source sounds, then sharpens them with a contrastive loss between clean and waveform-augmented versions. After that, a delayed bidirectional cross-attention aligns audio and visual features only in the later transformer layers, so the model relies on high-level instance semantics rather than low-level noise. On standard AVS and VPO benchmarks, DDAVS outperforms prior methods across single-, multi-, and multi-instance settings, with the largest gains in multi-source and semantic cases. The core claim is that grounding audio queries in a stable prototype space plus delaying cross-modal fusion is what drives the improvement.

Core claim

The central discovery is that a two-stage architecture—first disentangle audio into bank-grounded queries, then align them with vision through delayed bidirectional cross-attention—sets new state-of-the-art numbers on AVSBench and VPO. The audio query module uses a query transformer to compress the audio into a few learned queries, then refines them by cross-attending to a prototype memory bank of class centroids from single-source audio, anchoring each query to a stable semantic point. A contrastive optimization module applies a contrastive loss between queries from clean and augmented waveforms, pulling together corresponding query slots and pushing apart different slots, which the authors

What carries the argument

The central machinery is the combination of a bank-grounded audio query module (AQM), a contrastive optimization module (COM), and a delayed bidirectional audio-visual alignment module (AVAM). The AQM uses a query transformer to distill audio into a few learned queries, then refines them by cross-attending to a fixed prototype memory bank built from K-means centroids of single-source class embeddings, anchoring each query to a stable semantic anchor. The COM applies a contrastive loss between clean and waveform-augmented queries, treating query slot i from the clean audio as the positive of slot i from the augmented audio and all other slots as negatives. The AVAM performs two cross-attentio

Load-bearing premise

The contrastive loss assumes that the i-th audio query always encodes the same sound source after waveform augmentation, so that query slot i from the clean audio and query slot i from the augmented audio form a positive pair; if the query generator reorders or reassigns slots under perturbation, the loss would misalign unrelated queries and could harm disentanglement.

What would settle it

Take a trained DDAVS model, feed the same audio waveform both clean and augmented, and extract the n query vectors from each branch. Compute the pairwise cosine-similarity matrix between clean queries and augmented queries; if the slot-correspondence assumption holds, the identity permutation should yield the highest total similarity. Alternatively, randomly permute the augmented queries before computing the contrastive loss during training and compare final segmentation J&F; if a permuted pairing performs as well or better, slot identity is not essential and the assumed correspondence is unne

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If audio queries are truly anchored to a stable prototype space, the model should generalize to unseen mixtures more reliably than methods that derive semantics from nearest neighbors of the query audio alone, because the bank provides class-consistent anchors.
  • Delayed fusion implies that early layers should process unimodal, low-level features and that injecting audio earlier can hurt performance; the ablation table supports this by showing that injection into blocks 3 and 4 is the best configuration.
  • Because the contrastive loss is effective only when queries are bank-grounded, the disentanglement benefit depends on the prototype space; without AQM, using COM alone is unstable, as the paper notes.
  • The architecture's segmentation accuracy peaks at five audio queries and degrades beyond, suggesting the capacity of the query slots to represent distinct sound sources is limited around that number.
  • The method runs at 227 FPS with a modest increase in FLOPs, meaning the accuracy gains do not come at a prohibitive computational cost and the approach is practical for near-real-time applications.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The contrastive loss assumes that the i-th audio query always encodes the same sound source after waveform augmentation, so that query slot i from the clean audio and query slot i from the augmented audio form a positive pair; if the query generator reorders or reassigns slots under perturbation, the loss would misalign unrelated queries and could harm disentanglement rather than improve it.
  • The prototype bank is built from class centroids and kept fixed during training; an online or adaptive bank might better handle novel classes or domain shifts, which would be a testable extension.
  • The delayed bidirectional alignment can be seen as a curriculum that gradually increases cross-modal coupling; a simpler training schedule that ramps up the strength of cross-attention over layers might achieve similar gains, which would isolate the contribution of the delay mechanism.
  • The improvements on semantic AVS suggest the method could extend to open-vocabulary or zero-shot settings where the bank uses unsupervised clustering instead of class-labeled centroids, enabling segmentation of sounds not seen during training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DDAVS, a two-stage audio-visual segmentation framework. In the first stage, an Audio Query Module (AQM) uses Q-Former learnable queries to extract audio semantics and anchors them to a fixed prototype memory bank via cross-attention. A Contrastive Optimization Module (COM) applies an InfoNCE loss between queries derived from clean and waveform-augmented audio. In the second stage, an Audio-Visual Alignment Module (AVAM) performs delayed bidirectional cross-attention between audio queries and visual features, followed by a lightweight decoder. The paper reports state-of-the-art J&F scores on AVSBench (92.4 on S4, 75.1 on MS3, 52.6 on Semantic) and VPO (76.11 on MS, 72.84 on MSMI), with component ablations, efficiency statistics, and qualitative analyses.

Significance. If the reported results are reproducible, DDAVS is a meaningful advance for multi-source and multi-instance audio-visual segmentation. The bank-grounded query design and the delayed bidirectional alignment are well-motivated and clearly described. The paper includes component ablations, backbone variation experiments, fusion-position studies, efficiency analysis, and qualitative failure-case discussions, which strengthen the empirical contribution. However, the contrastive module's loss relies on an unvalidated slot-correspondence assumption, and key hyperparameters appear to be selected on the same benchmarks used for the final SOTA claim; both issues need to be addressed before the contribution is fully established.

major comments (3)
  1. [Sec. 3.2, Eq. (6)] The InfoNCE loss treats the i-th clean query and the i-th augmented query as positives. This assumes the Q-Former preserves slot identity across waveform perturbations: slot 5 before augmentation and slot 5 after augmentation should correspond to the same sound source. Since the query generator is a set of learned slots, nothing in the architecture or training objective enforces this correspondence. If slots permute under augmentation, the loss would pull together unrelated semantics and push apart related ones, potentially harming disentanglement rather than improving it. The paper provides no diagnostic for this assumption; the t-SNE visualization is qualitative, and the ablations only show that adding COM helps on average (Table 3: 70.89 to 73.47 on MS3). This is load-bearing because COM is one of the three core contributions. Please provide a quantitative test of slot correspondence
  2. [Sec. 4.4, Tables 4 and Fig. 6] The number of audio queries n=5 and the fusion position blocks 3-4 are selected by maximizing J&F on AVS-MS3 and AVSS. There is no statement that these ablations were performed on a held-out validation split; if they were run on the same test benchmarks used to report the final SOTA, the reported improvements may be inflated by selection. This is a serious concern because the final numbers in Tables 1 and 2 are the paper's central claim. Please clarify the split used for model selection, or report nested cross-validation results. At a minimum, report multiple random seeds with standard deviations, as the differences over prior methods are only 1.1-3.5 J&F points.
  3. [Tables 1 and 2] All benchmark numbers are single-run scores with no error bars or significance tests. Given that the claimed improvements over the previous best are modest in several settings (e.g., 1.3 on S4, 1.7 on AVSS, 1.81 on VPO-MS), the robustness of the SOTA claim is unclear. If the test sets are small or the variance across runs is non-negligible, the observed gaps may not be statistically meaningful. Reporting variances across at least three runs would substantially increase confidence in the empirical claims.
minor comments (5)
  1. [Fig. 7 / Fig. 8] Section 4.5 refers to the t-SNE figure as Fig. 8, but the figure is numbered Fig. 7 in the main text; the supplementary also has a Fig. 8 with t-SNE. Renumber to avoid ambiguity.
  2. [Sec. 3.3] The text says delayed cross-modal fusion is applied 'exclusively between the third and fourth layers', but Table 4 indicates injection at blocks 3 and 4. Please reword to avoid confusion between layer indices and block numbers.
  3. [Supplementary Tab. 7] The reverb parameter r is given as [20,40] without units. Specify the unit (e.g., RT60 in milliseconds) for reproducibility.
  4. [Supplementary Sec. B.2] The statement that prototype-as-positive, cross-sample, and raw-token contrastive variants were 'less stable or weaker' is not supported by numbers. Adding a small table or removing the claim would improve the paper.
  5. [Title / Abstract] The abstract uses 'Delayed Bidirectional Alignment via Disentangled Audio Semantics' while the title reads 'Disentangled Audio Semantics and Delayed Bidirectional Alignment'. Make the ordering consistent.

Circularity Check

0 steps flagged

No significant circularity; the SOTA claims are empirical results on external benchmarks and the architecture does not reduce to its inputs.

full rationale

DDAVS's central claims are benchmark numbers (Tables 1 and 2) against external AVSBench and VPO datasets. The derivation chain — AQM (Eqs. 1, 4), COM (Eqs. 5–6), AVAM (Eqs. 7–8), and the total loss (Eq. 9) — nowhere assumes those benchmark results; the losses are supervised by ground-truth masks and a contrastive objective on augmented audio queries. The prototype memory bank is explicitly constructed following DDESeg [28], an external prior work, so this is independent support rather than a self-citation chain. The same-author citations in Related Work (e.g., [11, 46, 53, 54]) are contextual and not load-bearing for the method's validity. The COM InfoNCE loss in Eq. (6) does rely on index-based pairing between clean and augmented query slots, but that is an unvalidated modeling assumption, not a circular reduction: the loss does not define the benchmark outcomes or the reported gains. Likewise, choosing n=5 and the injection position by ablations on the same benchmarks is hyperparameter tuning, not a fitted parameter renamed as a prediction. I find no equation or argument in the paper that is equivalent to its own input by construction, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical or formal entities. Its load-bearing baggage is empirical: design hyperparameters selected on the evaluation benchmarks, an untested slot-correspondence assumption in the contrastive loss, and reliance on external pretrained encoders and benchmark annotations.

free parameters (5)
  • number of audio queries n = 5
    Selected by scanning n=1..70 on AVS-MS3/AVSS (Fig. 6); the chosen value optimizes the same benchmarks used for the SOTA claim.
  • fusion blocks for delayed alignment = blocks 3 and 4
    Chosen by ablation on AVS-MS3/AVSS (Tab. 4), where the (3,4) combination performed best.
  • contrastive temperature tau = not reported
    Appears in Eq. (6); its value is not stated in the paper or supplement.
  • bank-refinement scale gamma = not reported
    Appears in Eq. (4); no value or sensitivity analysis is reported.
  • loss weights lambda_ce, lambda_dice, lambda_iou, lambda_con = not reported
    Sec. 3.4 says coefficients are detailed in the supplementary, but Supp. F only gives loss formulas, not the coefficients.
axioms (4)
  • domain assumption The prototype memory bank built from clean single-source audio remains a valid, fixed semantic anchor for arbitrary test mixtures.
    Supp. B builds the bank following DDESeg and keeps it fixed during training and inference; if test mixtures contain modes unseen in the bank, grounding could bias the queries.
  • ad hoc to paper The i-th query slot in the clean branch and the i-th query slot in the augmented branch correspond to the same sound source.
    Eq. (5)-(6) define positives by slot index; the paper does not justify this correspondence across waveform augmentations.
  • domain assumption HTSAT/AudioSet and MiT-B5/ImageNet pretrained features transfer to AVS benchmarks.
    Sec. 4.1 relies on pretrained encoders; Tab. 5 ablates backbone choices but does not establish that pretraining is necessary.
  • domain assumption AVSBench and VPO annotations and official evaluation protocols are trustworthy and comparable across methods.
    All SOTA claims depend on these benchmarks; the paper provides no label-noise analysis or cross-protocol validation.

pith-pipeline@v1.3.0-alltime-deepseek · 18026 in / 8575 out tokens · 88931 ms · 2026-08-03T14:27:08.021637+00:00 · methodology

0 comments
read the original abstract

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/

Figures

Figures reproduced from arXiv: 2512.20117 by Haoji Zhang, Isaac Ning Lee, Jingqi Tian, Jingxuan Niu, Tianrui Zhu, Xulong Bai, Yansong Tang, Yiheng Du, Yuji Wang.

Figure 1
Figure 1. Figure 1: Qualitative comparison of our DDAVS model and previous methods. DDAVS consistently outperforms previous approaches in challenging scenarios involving multiple classes, multiple sources, small or distant sound sources, and off-screen audio cues. Abstract Audio–Visual Segmentation (AVS) aims to localize sound￾producing objects at the pixel level by jointly leverag￾ing auditory and visual information. However… view at source ↗
Figure 2
Figure 2. Figure 2: For Audio Disentanglement, prior methods (a) use learned queries for semantics [6, 24, 47] or (b) derive disentangled features from K-nearest classes [28]. In contrast, our method (c) uses an audio prototype memory bank to ground audio queries, coupled with contrastive optimization to enhance their discriminability and robustness. For Audio-Visual Alignment, existing methods either (d) treat audio features… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the DDAVS framework. (a) The Audio Query Module (AQM) encodes original and augmented waveforms into disentangled semantic queries anchored to a prototype memory bank. (b) The Contrastive Optimization Module (COM) enhances query robustness through contrastive learning, used only during training. (c) The Audio-Visual Alignment Module (AVAM) fuses audio queries with visual features via stacked ali… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results on AVSBench-Semantic. DDAVS produces cleaner and more source-consistent masks than previous baselines AVSegFormer, AAVS, and DDESeg, especially in complex multi-source scenes with non-sounding distractors (horse–human), multiple active instruments (piano–guzheng), or multi-person guitar performances [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of attention maps of audio-injected transformers blocks at different layers. It is observed that in￾jecting audio features into blocks 3 and 4 bringing clearer instance￾level attention, compared to the blurry pattern at earlier blocks. 4.4. Ablation Study Framework Components. Tab. 3 analyzes the contribu￾tion of AQM, COM, and AVAM on AVS-MS3 and AVSS. The baseline only contains encoders, tra… view at source ↗
Figure 6
Figure 6. Figure 6: Effect of the number of audio queries. Performance on AVS-MS3 (top) and AVSS (bottom) as the number of audio queries n varies, where our choice n = 5 (ours) achieves the best overall results before larger n leads to performance degradation due to redundant queries. information into transformer block at different layers result in essential different attention pattern. As shown in [PITH_FULL_IMAGE:figures/f… view at source ↗
Figure 8
Figure 8. Figure 8: t-SNE visualization of audio embeddings and pro￾totype anchors. Class-wise prototypes consistently lie in high￾density regions and span multiple acoustic modes, demonstrating that the memory bank captures compact yet diverse semantic pat￾terns for guiding query refinement. Aa and A′ a are fed into the same audio encoder faud to pro￾duce latent features Ea = faud(Aa) and E′ a = faud(A′ a ), corresponding to… view at source ↗
Figure 9
Figure 9. Figure 9: Failure Cases for DDAVS. Left: low-light or visually [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Multi-class sources. DDAVS produces cleaner separation between different sound categories and avoids cross-class leakage. Baseline methods often mix activations across instruments or people, while DDAVS preserves clear boundaries for each sounding class. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Multi-instance sources. DDAVS maintains instance-level distinctions when several objects of the same class are active. Competing methods tend to merge nearby instances or miss smaller ones, while DDAVS keeps each sounding instance well separated. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Small or distant sources. DDAVS detects small or faraway sound emitters with higher spatial precision. Other models often produce incomplete or fragmented masks, whereas DDAVS retains accurate localization even when visual cues are weak. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Off-screen sources. DDAVS suppresses activations caused by off-screen audio and focuses on visible sound-producing regions. Previous methods frequently hallucinate masks in empty areas, while DDAVS keeps responses consistent with the visual scene. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation

    cs.CV 2026-05 conditional novelty 6.0

    SAFE-Pruner forecasts deep-layer visual-token saliency from historical attention maps and refreshes at subtask boundaries, enabling up to 1.89x faster VLA inference with minimal success-rate drop.

  2. SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation

    cs.CV 2026-05 unverdicted novelty 5.0

    SAFE-Pruner forecasts deep-layer token saliency in VLA models via semantic attention consistency and adaptive subtask detection to achieve up to 1.89x speedup with under 1.7% success rate loss.

  3. Segment Anything with Motion, Geometry, and Semantic Adaptation for Complex Nonlinear Visual Object Tracking

    cs.CV 2026-05 unverdicted novelty 5.0

    SAMOSA adapts SAM 2 for complex visual object tracking by integrating explicit nonlinear motion prediction, semantic cues for failure recovery, and geometric constraints for stability, outperforming prior SAM 2-based ...

Reference graph

Works this paper leans on

59 extracted references · 11 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Self-calibrated clip for training-free open-vocabulary segmentation.arXiv preprint arXiv:2411.15869, 2024

    Sule Bai, Yong Liu, Yifei Han, Haoji Zhang, and Yansong Tang. Self-calibrated clip for training-free open-vocabulary segmentation.arXiv preprint arXiv:2411.15869, 2024. 3

    arXiv 2024

  2. [2]

    Hts-at: A hierarchi- cal token-semantic audio transformer for sound classifica- tion and detection

    Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchi- cal token-semantic audio transformer for sound classifica- tion and detection. InProceedings of the IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing, pages 646–650, 2022. 6

    2022

  3. [3]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on ma- chine learning, pages 1597–1607. PmLR, 2020. 3

    2020

  4. [4]

    Unraveling in- stance associations: A closer look for audio-visual segmenta- tion

    Yuanhong Chen, Yuyuan Liu, Hu Wang, Fengbei Liu, Chong Wang, Helen Frazer, and Gustavo Carneiro. Unraveling in- stance associations: A closer look for audio-visual segmenta- tion. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 26497–26507,

  5. [5]

    Crab: A unified audio-visual scene understanding model with explicit cooperation

    Henghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, and Di Hu. Crab: A unified audio-visual scene understanding model with explicit cooperation. InProceed- ings of the Computer Vision and Pattern Recognition Con- ference, pages 18804–18814, 2025. 2

    2025

  6. [6]

    Avsegformer: Audio-visual segmentation with trans- former

    Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio-visual segmentation with trans- former. InProceedings of the AAAI Conference on Artificial Intelligence, pages 12155–12163, 2024. 2, 3, 5, 6

    2024

  7. [7]

    Audioset: An ontology and human- labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audioset: An ontology and human- labeled dataset for audio events. InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 776–780. IEEE, 2017. 6

    2017

  8. [8]

    Avs-mamba: Ex- ploring temporal and multi-modal mamba for audio-visual segmentation.IEEE Transactions on Multimedia, 2025

    Sitong Gong, Yunzhi Zhuge, Lu Zhang, Yifan Wang, Ping- ping Zhang, Lijun Wang, and Huchuan Lu. Avs-mamba: Ex- ploring temporal and multi-modal mamba for audio-visual segmentation.IEEE Transactions on Multimedia, 2025. 3

    2025

  9. [9]

    Complementary and contrastive learning for audio-visual segmentation.IEEE Transactions on Multime- dia, 2025

    Sitong Gong, Yunzhi Zhuge, Lu Zhang, Pingping Zhang, and Huchuan Lu. Complementary and contrastive learning for audio-visual segmentation.IEEE Transactions on Multime- dia, 2025. 3

    2025

  10. [10]

    Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020. 3

    2020

  11. [11]

    Thinking with bounding boxes: Enhanc- ing spatio-temporal video grounding via reinforcement fine- tuning.arXiv preprint arXiv:2511.21375, 2025

    Xin Gu, Haoji Zhang, Qihang Fan, Jingxuan Niu, Zhipeng Zhang, Libo Zhang, Guang Chen, Fan Chen, Longyin Wen, and Sijie Zhu. Thinking with bounding boxes: Enhanc- ing spatio-temporal video grounding via reinforcement fine- tuning.arXiv preprint arXiv:2511.21375, 2025. 3

    arXiv 2025

  12. [12]

    Improving audio-visual segmenta- tion with bidirectional generation

    Dawei Hao, Yuxin Mao, Bowen He, Xiaodong Han, Yuchao Dai, and Yiran Zhong. Improving audio-visual segmenta- tion with bidirectional generation. InProceedings of the AAAI Conference on Artificial Intelligence, pages 2067– 2075, 2024. 5

    2067

  13. [13]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 9729–9738, 2020. 3

    2020

  14. [14]

    Cnn archi- tectures for large-scale audio classification

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn archi- tectures for large-scale audio classification. InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 131–135. IEEE, 2017. 12, 13

    2017

  15. [15]

    Discovering sound- ing objects by audio queries for audio visual segmentation

    Shaofei Huang, Han Li, Yuqing Wang, Hongji Zhu, Jiao Dai, Jizhong Han, Wenge Rong, and Si Liu. Discovering sound- ing objects by audio queries for audio visual segmentation. InProceedings of the International Joint Conference on Ar- tificial Intelligence, 2023. 5

    2023

  16. [16]

    Re- visiting audio-visual segmentation with vision-centric trans- former

    Shaofei Huang, Rui Ling, Tianrui Hui, Hongyu Li, Xu Zhou, Shifeng Zhang, Si Liu, Richang Hong, and Meng Wang. Re- visiting audio-visual segmentation with vision-centric trans- former. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 8352–8361, 2025. 3, 5

    2025

  17. [17]

    Speech simclr: Combining contrastive and reconstruction objective for self-supervised speech representation learning

    Dongwei Jiang, Wubo Li, Miao Cao, Wei Zou, and Xiangang Li. Speech simclr: Combining contrastive and reconstruction objective for self-supervised speech representation learning. arXiv preprint arXiv:2010.13991, 2020. 5

    Pith/arXiv arXiv 2010

  18. [18]

    Segment anything in high qual- ity.Advances in Neural Information Processing Systems, 36: 29914–29934, 2023

    Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi- Keung Tang, Fisher Yu, et al. Segment anything in high qual- ity.Advances in Neural Information Processing Systems, 36: 29914–29934, 2023. 1

    2023

  19. [19]

    Data augmenting contrastive learning of speech representations in the time domain

    Eugene Kharitonov, Morgane Rivi `ere, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazar ´e, Matthijs Douze, and Emmanuel Dupoux. Data augmenting contrastive learning of speech representations in the time domain. In2021 IEEE Spoken Language Technology Workshop (SLT), pages 215–

  20. [20]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 1

    2023

  21. [21]

    From waveforms to pixels: A survey on audio-visual segmentation.arXiv preprint arXiv:2508.03724, 2025

    Jia Li and Yapeng Tian. From waveforms to pixels: A survey on audio-visual segmentation.arXiv preprint arXiv:2508.03724, 2025. 3

    Pith/arXiv arXiv 2025

  22. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. 4

    2023

  23. [23]

    Catr: Combinatorial-dependence audio-queried transformer for audio-visual video segmentation

    Kexin Li, Zongxin Yang, Lei Chen, Yi Yang, and Jun Xiao. Catr: Combinatorial-dependence audio-queried transformer for audio-visual video segmentation. InProceedings of the 31st ACM International Conference on Multimedia, pages 1485–1494, 2023. 3, 5

    2023

  24. [24]

    Qdformer: Towards ro- bust audiovisual segmentation in complex environments with quantization-based semantic decomposition

    Xiang Li, Jinglu Wang, Xiaohao Xu, Xiulian Peng, Rita Singh, Yan Lu, and Bhiksha Raj. Qdformer: Towards ro- bust audiovisual segmentation in complex environments with quantization-based semantic decomposition. InProceedings 9 of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 3402–3413, 2024. 2, 3, 5

    2024

  25. [25]

    Audio-visual seg- mentation by exploring cross-modal mutual semantics

    Chen Liu, Peike Patrick Li, Xingqun Qi, Hu Zhang, Lincheng Li, Dadong Wang, and Xin Yu. Audio-visual seg- mentation by exploring cross-modal mutual semantics. In Proceedings of the 31st ACM International Conference on Multimedia, pages 7590–7598, 2023. 3, 5

    2023

  26. [26]

    Bavs: Bootstrapping audio- visual segmentation by integrating foundation knowledge

    Chen Liu, Peike Li, Hu Zhang, Lincheng Li, Zi Huang, Dadong Wang, and Xin Yu. Bavs: Bootstrapping audio- visual segmentation by integrating foundation knowledge. IEEE Transactions on Multimedia, 2024. 5

    2024

  27. [27]

    Robust audio-visual segmentation via audio- guided visual convergent alignment

    Chen Liu, Peike Li, Liying Yang, Dadong Wang, Lincheng Li, and Xin Yu. Robust audio-visual segmentation via audio- guided visual convergent alignment. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 28922–28931, 2025. 3, 6

    2025

  28. [28]

    Dynamic derivation and elimination: Audio visual segmentation with enhanced audio semantics

    Chen Liu, Liying Yang, Peike Li, Dadong Wang, Lincheng Li, and Xin Yu. Dynamic derivation and elimination: Audio visual segmentation with enhanced audio semantics. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3131–3141, 2025. 1, 2, 3, 5, 6

    2025

  29. [29]

    Audio-aware query-enhanced trans- former for audio-visual segmentation.arXiv preprint arXiv:2307.13236, 2023

    Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, and Ya Zhang. Audio-aware query-enhanced trans- former for audio-visual segmentation.arXiv preprint arXiv:2307.13236, 2023. 1, 3, 5

    Pith/arXiv arXiv 2023

  30. [30]

    Open-vocabulary segmentation with semantic-assisted calibration

    Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3491– 3500, 2024

    2024

  31. [31]

    Stepping out of similar seman- tic space for open-vocabulary segmentation

    Yong Liu, Song-Li Wu, Sule Bai, Jiahao Wang, Yitong Wang, and Yansong Tang. Stepping out of similar seman- tic space for open-vocabulary segmentation. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 22664–22674, 2025. 3

    2025

  32. [32]

    Tavis: Text-bridged audio- visual segmentation with foundation models.arXiv preprint arXiv:2506.11436, 2025

    Ziyang Luo, Nian Liu, Xuguang Yang, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shah- baz Khan, and Junwei Han. Tavis: Text-bridged audio- visual segmentation with foundation models.arXiv preprint arXiv:2506.11436, 2025. 3, 5

    arXiv 2025

  33. [33]

    Consistency-queried transformer for audio-visual segmentation.IEEE Transac- tions on Image Processing, 2025

    Ying Lv, Zhi Liu, and Xiaojun Chang. Consistency-queried transformer for audio-visual segmentation.IEEE Transac- tions on Image Processing, 2025. 3

    2025

  34. [34]

    Step- ping stones: A progressive training strategy for audio-visual semantic segmentation.arXiv preprint arXiv:2407.11820,

    Juncheng Ma, Peiwen Sun, Yaoting Wang, and Di Hu. Step- ping stones: A progressive training strategy for audio-visual semantic segmentation.arXiv preprint arXiv:2407.11820,

    Pith/arXiv arXiv

  35. [35]

    Multimodal variational auto-encoder based audio-visual segmentation

    Yuxin Mao, Jing Zhang, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Multimodal variational auto-encoder based audio-visual segmentation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 954– 965, 2023. 5

    2023

  36. [36]

    Contrastive conditional latent diffusion for audio-visual segmentation.IEEE Transactions on Image Processing, 2025

    Yuxin Mao, Jing Zhang, Mochu Xiang, Yunqiu Lv, Dong Li, Yiran Zhong, and Yuchao Dai. Contrastive conditional latent diffusion for audio-visual segmentation.IEEE Transactions on Image Processing, 2025. 3, 5

    2025

  37. [37]

    Weakly-supervised audio- visual segmentation.Advances in Neural Information Pro- cessing Systems, 36:17208–17221, 2023

    Shentong Mo and Bhiksha Raj. Weakly-supervised audio- visual segmentation.Advances in Neural Information Pro- cessing Systems, 36:17208–17221, 2023. 3

    2023

  38. [38]

    Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 3

    Pith/arXiv arXiv 2018

  39. [39]

    Extending segment anything model into au- ditory and temporal dimensions for audio-visual segmenta- tion

    Juhyeong Seon, Woobin Im, Sebin Lee, Jumin Lee, and Sung-Eui Yoon. Extending segment anything model into au- ditory and temporal dimensions for audio-visual segmenta- tion. In2024 IEEE International Conference on Image Pro- cessing (ICIP), pages 2480–2486. IEEE, 2024. 3

    2024

  40. [40]

    Cross-modal cognitive consensus guided audio-visual segmentation.IEEE Transactions on Multime- dia, 2024

    Zhaofeng Shi, Qingbo Wu, Fanman Meng, Linfeng Xu, and Hongliang Li. Cross-modal cognitive consensus guided audio-visual segmentation.IEEE Transactions on Multime- dia, 2024. 3

    2024

  41. [41]

    video-salmonn: Speech-enhanced audio-visual large language models.arXiv preprint arXiv:2406.15704, 2024

    Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models.arXiv preprint arXiv:2406.15704, 2024. 2

    Pith/arXiv arXiv 2024

  42. [42]

    Unveiling and mitigating bias in audio visual segmentation.arXiv preprint arXiv:2407.16638, 2024

    Peiwen Sun, Honggang Zhang, and Di Hu. Unveiling and mitigating bias in audio visual segmentation.arXiv preprint arXiv:2407.16638, 2024. 3, 5, 6

    Pith/arXiv arXiv 2024

  43. [43]

    Pvt v2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational visual media, 8(3):415–424, 2022. 12, 13

    2022

  44. [44]

    Prompting segmentation with sound is gen- eralizable audio-visual source localizer

    Yaoting Wang, Weisong Liu, Guangyao Li, Jian Ding, Di Hu, and Xi Li. Prompting segmentation with sound is gen- eralizable audio-visual source localizer. InProceedings of the AAAI Conference on Artificial Intelligence, pages 5669– 5677, 2024. 3, 5

    2024

  45. [45]

    Ref-avs: Refer and segment objects in audio-visual scenes.European Conference on Computer Vision, 2024

    Yaoting Wang, Peiwen Sun, Dongzhan Zhou, Guangyao Li, Honggang Zhang, and Di Hu. Ref-avs: Refer and segment objects in audio-visual scenes.European Conference on Computer Vision, 2024. 2

    2024

  46. [46]

    Ponder & press: Advancing visual gui agent towards general computer control

    Yiqin Wang, Haoji Zhang, Jingqi Tian, and Yansong Tang. Ponder & press: Advancing visual gui agent towards general computer control. InFindings of the Association for Com- putational Linguistics: ACL 2025, pages 1461–1473, 2025. 3

    2025

  47. [47]

    Avesformer: Efficient trans- former design for real-time audio-visual segmentation.arXiv preprint arXiv:2408.01708, 2024

    Zili Wang, Qi Yang, Linsu Shi, Jiazhong Yu, Qinghua Liang, Fei Li, and Shiming Xiang. Avesformer: Efficient trans- former design for real-time audio-visual segmentation.arXiv preprint arXiv:2408.01708, 2024. 2, 3, 5

    Pith/arXiv arXiv 2024

  48. [48]

    Segformer: Simple and efficient design for semantic segmentation with transform- ers.Advances in Neural Information Processing Systems, 34:12077–12090, 2021

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers.Advances in Neural Information Processing Systems, 34:12077–12090, 2021. 6, 7

    2021

  49. [49]

    Cooperation does matter: Exploring multi-order bilateral relations for audio- visual segmentation

    Qi Yang, Xing Nie, Tong Li, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, and Shiming Xiang. Cooperation does matter: Exploring multi-order bilateral relations for audio- visual segmentation. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 27134–27143, 2024. 3

    2024

  50. [50]

    Towards omnimodal expressions and reasoning in 10 referring audio-visual segmentation

    Kaining Ying, Henghui Ding, Guangquan Jie, and Yu-Gang Jiang. Towards omnimodal expressions and reasoning in 10 referring audio-visual segmentation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22575–22585, 2025. 3

    2025

  51. [51]

    Implicit counterfactual learning for audio-visual segmentation

    Mingfeng Zha, Tianyu Li, Guoqing Wang, Peng Wang, Yangyang Wu, Yang Yang, and Heng Tao Shen. Implicit counterfactual learning for audio-visual segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22349–22360, 2025. 3, 5

    2025

  52. [52]

    Videollama 3: Frontier multi- modal foundation models for image and video understand- ing.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multi- modal foundation models for image and video understand- ing.arXiv preprint arXiv:2501.13106, 2025. 2

    Pith/arXiv arXiv 2025

  53. [53]

    Thinking with videos: Multimodal tool- augmented reinforcement learning for long video reasoning

    Haoji Zhang, Xin Gu, Jiawen Li, Chixiang Ma, Sule Bai, Chubin Zhang, Bowen Zhang, Zhichao Zhou, Dongliang He, and Yansong Tang. Thinking with videos: Multimodal tool- augmented reinforcement learning for long video reasoning. arXiv preprint arXiv:2508.04416, 2025. 3

    Pith/arXiv arXiv 2025

  54. [54]

    Flash-vstream: Efficient real- time understanding for long video streams.arXiv preprint arXiv:2506.23825, 2025

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Ji- ashi Feng, and Xiaojie Jin. Flash-vstream: Efficient real- time understanding for long video streams.arXiv preprint arXiv:2506.23825, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    Alignedgen: Aligning style across generated images.arXiv preprint arXiv:2509.17088, 2025

    Jiexuan Zhang, Yiheng Du, Qian Wang, Weiqi Li, Yu Gu, and Jian Zhang. Alignedgen: Aligning style across generated images.arXiv preprint arXiv:2509.17088, 2025. 3

    arXiv 2025

  56. [56]

    Audio-visual segmentation

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. InEuropean Conference on Computer Vision, pages 386–

  57. [57]

    Audio-visual segmentation with semantics.International Journal of Computer Vision,

    Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Ling- peng Kong, Meng Wang, et al. Audio-visual segmentation with semantics.International Journal of Computer Vision,

  58. [58]

    Yang-Hao Zhou, Heyan Huang, Cunhan Guo, Rong-Cheng Tu, Zeyu Xiao, Bo Wang, and Xian-Ling Mao. Aloha: Adapting local spatio-temporal context to enhance the audio- visual semantic segmentation.ACM Transactions on Mul- timedia Computing, Communications and Applications, 21 (6):1–23, 2025. 3 11 DDA VS: Disentangled Audio Semantics and Delayed Bidirectional Al...

    2025

  59. [403]

    2, 3, 5, 6

    Springer, 2022. 2, 3, 5, 6

    2022