Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation

Huchuan Lu; Jiahong Jiang; Jialong Zhong; Jingjing Li; Kai Peng; Leiye Liu; Miao Zhang; TingWei Liu; Wei Ji; Yongri Piao

REVIEW 4 major objections 5 minor 50 references

Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that disentangling mixed audio into separate streams before matching them to video pixels, then using audio dynamics to modulate the state-transition parameter of a Mamba tracker, enables state-of-the-art audio-visual insta

desk verdict Plausible new SOTA on AVISeg with two genuinely new mechanisms, but the headline 7.8% compares against the wrong baseline and Eq. (3) has a shape error that needs fixing. read the letter →

arxiv 2608.03264 v1 pith:PM3PES4P submitted 2026-08-04 cs.MM cs.CVcs.SD

Leiye Liu , Miao Zhang , Jiahong Jiang , Jingjing Li , Jialong Zhong , Kai Peng , Tingwei Liu , Wei Ji

show 2 more authors

Yongri Piao Huchuan Lu

This is my paper

classification cs.MMcs.CVcs.SD

keywords audio-visualinstancesegmentationacousticdisentanglementsourceseparationasynchronousdynamicsstatespacemodelsMambacross-modalcorrespondencevideo

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles audio-visual instance segmentation: producing a pixel-level mask for each sounding object and holding its identity across time. It argues that the two bottlenecks—mixed audio that superimposes several sources in one channel, and audio and video states that change at different moments—can be solved by separating the audio before matching, and by letting audio dynamics control a state-space tracker's transition speed. The proposed H2S combines the Acoustic-Semantic Projector (ASP), which uses a frozen MixIT separator and VGGish to create independent audio streams and matches them to video via k-means cluster attention, with the Asynchronous Dynamics Modulator (ADM), which injects audio-derived gating into Mamba's Δ parameter. On the AVISeg benchmark, with a COCO-pretrained ResNet-50, H2S reports 48.54 mAP, 65.70 HOTA, and 47.58 FSLA, 7.8% above the previous best, and a 14.09 mAP gain over the fixed-window baseline on an asynchronous subset. If the claims hold, the work establishes a reusable recipe for audio-visual instance-level understanding: disentangle first, then let audio steer the temporal dynamics.

What carries the argument

The two load-bearing mechanisms are the Acoustic-Semantic Projector (ASP) and the Asynchronous Dynamics Modulator (ADM). ASP consists of Audio Source Disentanglement (ASD), a frozen MixIT separation model plus VGGish that converts one mixed audio channel into N_a=8 independent feature streams, and a Hierarchical Correspondence Mechanism (HCM) that k-means clusters audio and video tokens, scores cluster-centroid similarities in a shared semantic space, and uses top-p filtered sparse matrix multiplication to attend from audio streams to visual tokens. ADM wraps an Audio-Dynamically Modulated Mamba (AMM), where the SSM's Δ (the zero-order-hold timescale) is augmented by audio features passed th

What would settle it

Replace the frozen VGGish audio encoder with a different pretrained audio network (or with random projections) and retrain H2S on AVISeg; if the mAP gain over the no-ASP baseline largely persists, the cluster-correspondence mechanism does not depend on the shared semantic space. Separately, evaluate on a constructed test set with more than eight concurrently sounding objects: a sharp mAP drop would confirm the fixed N_a=8 streams as the binding limit.

Watch

Extended reading notes

Core claim

H2S is an audio-visual instance segmentation architecture whose central claim is that precise audio-visual correspondence and robust tracking under asynchronous state changes are both achievable by treating the audio track as a set of separable sources and by making the temporal model state-aware. The ASP first runs MixIT source separation and VGGish to produce N_a=8 independent audio streams (frozen), then k-means clusters audio and video features, computes a cluster-centroid similarity weighted by cluster size, filters it with top-p, and projects the result back to tokens to produce sparse cross-modal attention. The ADM feeds video object queries into a Mamba block and modifies the Δ param

Load-bearing premise

The load-bearing premise is that the frozen VGGish audio streams and the trainable video features are already comparable in a shared semantic space, so that the k-means centroid dot product in Eq. (3) measures genuine audio-visual correspondence; the paper describes no learned projection or semantic alignment, and the fixed eight-stream limit also caps the number of simultaneous sources the model can represent.

Editorial extensions

If this is right

On AVISeg, H2S establishes a new reported state of the art: 48.54 mAP with a COCO-pretrained ResNet-50, beating the AVISM baseline by 7.8% relative mAP and the strongest prior AVIS result by roughly 4%.
Explicit source separation before matching is the effective ingredient: naively concatenating MixIT-separated features to the AVISM baseline drops mAP by 0.62, while the full ASP adds +3.50 mAP, so the clustering-and-attention step is what converts separated audio into a gain.
The ADM's Δ modulation primarily buys tracking accuracy: switching from modulating Δ to modulating B or C reduces HOTA, and on the asynchronous subset H2S gains 14.09 mAP over the fixed-window AVISM baseline.
The performance gain does not come from the external YFCC100M pretraining of MixIT: retraining the separator on AVISeg alone gives 48.62 mAP, essentially identical to 48.54, which the paper reads as evidence that the architecture, not external data, drives the result.
Zero-shot transfer to the AVSS dataset improves class-agnostic J and F scores by 1.13 and 1.85 over AVISM, suggesting the learned audio-visual alignment transfers beyond the 26 AVISeg labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural testable extension is to check whether the cluster-level correspondence remains discriminative under a learned projection or contrastive loss; if the k-means centroids already work without alignment, training such a projection might push mAP further, and if they do not, the current 48.54 would be tied to the specific VGGish geometry.
The fixed N_a=8 separation streams and cluster counts (8) look tuned to the AVISeg distribution; in denser or more varied scenes an adaptive stream count would be needed, and one quick check is to measure mAP on synthetic scenes with more than eight simultaneous sources.
The authors scope the claim to offline inference; extending ADM to causal/streaming settings would require replacing full-context audio separation with a streaming separator and causal scanning, so the SOTA result should not be read as an online tracking result.
If the recipe generalizes, the same 'disentangle then cluster-match, then modulate timescale' pattern could apply to other many-to-one multimodal alignment problems, such as multi-speaker localization or audio-guided video object tracking, where one modality is a mixed superposition and the other is spatially structured.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a credible SOTA paper on AVISeg with two genuinely new modules, but the headline number is inflated by comparing to the wrong baseline, and Eq. (3) has a shape error that needs to be fixed before the mechanism can be fully trusted.

The genuinely new stuff is the combination of a MixIT-based sound source separation with a k-means cluster-to-cluster sparse attention (the ASP), and the idea of modulating Mamba's delta parameter with audio dynamics (the ADM). The ablation study is more careful than most in this area: they show that naive concatenation of separated audio actually hurts, that modulating delta beats modulating B or C, and that the MixIT pre-training dataset doesn't explain the gains. That is honest, reproducible-sounding engineering, and I believe the 48.54 mAP result is real.

The soft spots are real but fixable. First, Eq. (3) defines C_ij as a D×D outer product, then feeds it into a softmax over cluster indices. That can't be right; it should be a scalar dot product. The authors need to correct the notation. Second, the 'shared semantic space' claim is hand-wavy. VGGish features are frozen and pixel-decoder features are trained; the paper doesn't describe any projection or metric alignment between them. Since ASP accounts for +3.83 mAP, the authors need to convince us this isn't just added sparse attention capacity. They could show, for example, that the cluster assignments align with known sound classes. Third, the abstract's '7.8% over the previous best' only holds against AVISM, not ACVIS (46.68 → 48.54 is about 4.0%). That is misleading and should be fixed. There are also minor reporting issues (the text in A.1 says hierarchical Top-p is best, but Table 5 shows fixed 0.8 gives 48.67), and the asynchronous subset is never specified.

Overall, the core idea is worthwhile and the empirical work is above average for the subfield. A reviewer should get a chance to push on the correspondence mechanism and ask for code and error bars. That's a normal revision path, not a desk reject.

Referee Report

4 major / 5 minor

Summary. The paper proposes H2S (Hear to See), a two-module architecture for audio-visual instance segmentation (AVIS). An Acoustic-Semantic Projector (ASP) separates the mixed audio signal into independent streams with a frozen MixIT source-separation model and a frozen VGGish encoder, then performs a k-means-based Hierarchical Correspondence Mechanism (HCM) to associate audio and visual clusters. An Asynchronous Dynamics Modulator (ADM) adapts the Mamba state-transition parameter Δ using audio gating, intended to improve tracking through asynchronous audio-visual state changes. Experiments on AVISeg report 48.54 mAP with a COCO-pretrained ResNet50, which the abstract and introduction describe as surpassing the previous best by 7.8%. Ablations evaluate ASP, ADM, ASD, HCM hyperparameters, source-separation pretraining, and zero-shot generalization to AVSS.

Significance. If the reported numbers are correct, H2S would be a meaningful advance on AVISeg, and the design is original: disentangling audio sources with an explicit separation model and modulating Mamba's Δ with audio information are both fresh ideas in this task. The empirical study is unusually careful in several places: Table 7 shows that naively concatenating MixIT features to AVISM hurts performance, Table 8 attempts to rule out YFCC100M pretraining as the sole source of gain, and the asynchronous subset targets the paper's central difficulty. However, two load-bearing issues currently prevent acceptance: the formal definition of the HCM correlation is ill-posed, and the '7.8% over previous best' claim is numerically inaccurate with respect to the actual previous best (ACVIS). The hyperparameter narrative is also internally inconsistent with the authors' own Table 5. These issues are substantial but appear fixable within the scope of a major revision.

major comments (4)

[Abstract, §1, §4.3, Table 2] The claim of 'surpassing the previous best approach by 7.8%' is not correct against the strongest prior. Table 2 lists ACVIS with COCO-pretrained ResNet50 at 46.68 mAP; H2S reaches 48.54 mAP, a relative improvement of about 3.99%, not 7.8%. The 7.8% figure is obtained only when comparing to AVISM at 45.04 mAP. The abstract and introduction should be corrected so the comparator is explicitly ACVIS and the percentage is computed relative to the actual previous best.
[Eq. (3), §3.3.2, Table 3] The HCM correlation is not dimensionally defined. If centroid(AC_i) and centroid(VC_j) are both D-dimensional vectors, then centroid(AC_i) × centroid(VC_j)^T is a D×D outer product, and the subsequent expression P = VC_j exp(C_ij) / Σ_k VC_k exp(C_ik) is ill-formed. If '×' is intended to be a dot product, the result is a scalar only under that reading, but the text still does not justify why dot products between VGGish audio centroids and pixel-decoder video centroids are semantically meaningful. The paper asserts these live in 'a shared semantic space,' yet no learned projection, metric alignment, or semantic supervision is described. VGGish is trained for AudioSet tagging, while pixel-decoder features are trained for segmentation; their raw geometries are not commensurable. Because Table 3 attributes +3.83 mAP to the ASP, this is a load-bearing issue: the observed gain could come from
[Appendix A.1, Table 5] The textual interpretation of the hyperparameter ablation is contradicted by the table. The text says 'Setting the cluster number to 8 achieves the best performance,' but Table 5 (left) shows mAP = 48.87 for C_ka = C_kv = 10 versus 48.54 for 8. Similarly, the text states that the hierarchical Top-P configuration achieves optimal performance, but the uniform Top-P = 0.8 row has mAP = 48.67 versus 48.54 for the hierarchical [0.7, 0.8, 0.9] configuration. The authors should either correct the text, report which metric determines 'best,' or explain why the selected hyperparameters are preferred despite the higher mAP of the alternative settings.
[§4.6, Figure 8] The asynchronous-subset experiment is central to the ADM claim, but the subset is not specified: there is no description of how 'asynchronous' is defined, how many videos/frames it contains, or whether the subset is fixed and used identically for both methods. Moreover, the comparison is only against AVISM, not against ACVIS, the actual previous best. Without this information, the +14.09 mAP result is difficult to interpret, and the claim that ADM is responsible for the robustness gain is not fully established. Please provide the subset construction details and, if possible, include ACVIS as a comparison.

minor comments (5)

[Throughout] There are several typos and grammatical issues: 'multimodel' should be 'multimodal' (§2.2, §3), 'workes' should be 'works' (§3.2, §3.4), 'focuse' should be 'focused' (§2.2), and 'Intuitively, a larger Δ forces... NaN' has an extra period. Table 1 has '32.2229.83' (missing space) in the AVISM row.
[Eq. (3) and §3.3.2] The symbol D in the denominator sqrt(D) is not defined in Section 3.3.2. Also, the Top-P algorithm is never formally described; only the probability p is listed in Implementation Details. Please define the procedure or cite a source.
[§4.3] The sentence 'outperforms previous approaches, improving the mAP by 7.8%' should explicitly name ACVIS as the actual previous best, otherwise the comparison is misleading. The GPT-4-style claim 'state-of-the-art' is fine, but the numeric basis must be transparent.
[Fig. 9 and Appendix A.2] The zero-shot AVSS evaluation is only against AVISM. Since ACVIS is also an AVIS method, it would strengthen the generalization claim to include it. In addition, the two bar charts use different y-axis scales, which makes visual comparison of J and F gains less intuitive, though the numbers are stated in the text.
[§5 and Appendix B] The limitation about offline operation is well stated. However, the hard cap of N_a = 8 separated audio streams is not discussed as a limitation; a scene with more than eight simultaneous sources cannot be represented by the architecture. This should be acknowledged, or justified with dataset statistics.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central claim is an independently evaluated benchmark result, and no load-bearing step reduces to its own inputs by definition or self-citation.

full rationale

The paper's central claim is empirical: H2S achieves 48.54 mAP on the AVISeg test set, supported by comparisons and ablations against held-out test data. The ASP and ADM are evaluated through standard component-wise ablations (Tables 3-8), not derived from the definitions of the modules. Self-citations (e.g., [27], [33], [35], [43]) appear only in related-work or comparison contexts and are not load-bearing for the claimed result. The HCM's 'shared semantic space' assertion and the dimensional ambiguity of Eq. (3) are technical correctness concerns, not circularity: the k-means centroids are computed on the same features being matched, but the matching is trained end-to-end with supervision, so no fitted parameter is relabeled as a prediction. The paper's own limitation passages (Sec. 5 and Appendix B) candidly acknowledge missing online validation and incomplete separation-model comparisons, which further supports that the architecture is not being justified by an unverified self-citation chain. No circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 5 free parameters · 6 assumptions · 2 invented entities

The empirical architecture borrows Mamba discretization, MixIT, and VGGish from prior work; it contributes a non-differentiable k-means semantic correspondence and an audio-gated delta modulation. The shared-semantic-space premise and the fixed source-stream count are the main unverified inputs.

free parameters (5)

N_a = 8
Number of separated audio streams from MixIT. Chosen by hand; if a scene has more than eight simultaneous sources, the model cannot represent them.
C_ka and C_kv = 8
Number of k-means clusters for audio and video in HCM. Tuned in Table 5; the text says 8 is optimal, but Table 5 shows C=10 gives 48.87 mAP and C=[6,8,10] gives 48.63 mAP, both above 48.54.
Top-P values = [0.7, 0.8, 0.9] per stage
Thresholds for filtering importance scores in HCM. Tuned in Table 5; Top-P=0.8 alone gives 48.67 mAP, above the hierarchical setting's 48.54 mAP.
Loss weights = lambda_frame=1, lambda_video=1, lambda_sim=0.5
Weights in Eq. (6) are set by hand without sensitivity analysis.
Training schedule and clip length = LR=1e-4, decay at 32k, 48k iterations, batch 2, 5 frames
Standard but hand-chosen; final metrics depend on these choices.

assumptions (6)

standard math Zero-Order Hold discretization of SSM dynamics (Eq. 2) is valid and preserves the semantics of delta.
Borrowed from Mamba [12]; used in Eq. (5) to compute discretized A and B for ADM.
ad hoc to paper Audio and video features lie in a shared semantic space, so k-means centroids can be compared by dot product.
Eq. (3) computes C_ij = centroid(AC_i) * centroid(VC_j)^T / sqrt(D) with no learned projection. The paper asserts a shared semantic space without training or verification.
domain assumption Frozen MixIT separates overlapping audio into N_a independent streams that correspond to distinct sounding instances.
Section 3.3.1 uses MixIT pretrained on YFCC100M; N_a=8 is fixed. Tables 4, 7, and 8 give in-paper support but no guarantee for all AVISeg scenes.
domain assumption VGGish features from separated streams preserve enough instance-discriminative detail to distinguish sound sources.
Section 3.3.1 uses frozen VGGish; the paper does not show that VGGish features separate instances of the same category.
domain assumption Mamba's delta intuition (larger delta focuses on current input, smaller delta retains history) remains valid when delta is augmented by audio-derived gating.
ADM in Eq. (5) adds an audio-gated term to delta; the paper relies on this interpretation for tracking behavior but gives only one qualitative histogram in Figure 7.
domain assumption AVISeg test annotations and official split, and the prior methods' reported numbers, are reliable and comparable.
Required for the SOTA claim; no code or evaluation scripts are provided to verify.

invented entities (2)

Acoustic-Semantic Projector (ASP)
purpose: Disentangles mixed audio into streams and builds hierarchical semantic-to-spatial attention between audio and video.
Validated only by AVISeg ablations in this paper; no external evaluation isolates ASP.
Asynchronous Dynamics Modulator (ADM)
purpose: Modulates Mamba's delta using audio dynamics to adapt state transitions for asynchronous audio-visual tracking.
Validated only by AVISeg ablations and a single qualitative delta histogram; no external falsifiable test.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation." pith.science (2026). https://pith.science/paper/PM3PES4P

@misc{pith2026260803264,
  author       = {Pith},
  title        = {Pith review of: Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PM3PES4P}},
  note         = {Machine review of arXiv:2608.03264}
}

read the original abstract

Audio-visual instance segmentation (AVIS) requires accurately identifying and tracking individual sounding objects with pixel-level masks. Existing methods struggle to match overlapping acoustic events with visual instances and handle asynchronous audio-visual dynamics. Therefore, two critical questions arise: how can a model establish precise correspondence between overlapping sound sources and visual instances, and how can a model maintain robust tracking when audio and visual signals are temporally misaligned?This paper proposes Hear to See (H2S), addressing these challenges through two mechanisms. The Acoustic-Semantic Projector (ASP) disentangles mixed audio and establishes hierarchical correspondence from semantic to spatial domains. The Asynchronous Dynamics Modulator (ADM) adaptively adjusts state transitions via audio-modulated Mamba, prioritizing current information during dynamic variations and maintaining continuity in stable periods.Experiments on AVISeg show H2S achieves SOTA performance, attaining 48.54 mAP with a COCO pretrained ResNet50 and surpassing the previous by 7.8\%. The code will be open-sourced once the paper is accepted. The source code will be publicly available at https://github.com/leiyeliu/H2S.

Figures

Figures reproduced from arXiv: 2608.03264 by the authors.

**Figure 2.** Overall architecture of the proposed H2S. (a) The proposed Acoustic-Semantic Projector (ASP), consisting of Audio [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Structure of the Hierarchical Correspondence Mech [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Structure of the Asynchronous Dynamics Modulator (ADM). (a) Overall structure of ADM. (b) Structure of Audio [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Visualization of segmentation results. Each group contains three rows: the first row shows the ground truth, the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 7.** Figure 7: Visualization of the dynamic modulation Δ and predictions under asynchronous audio-visual events. (a) Frame distribution histogram of the averaged Δ. (b) The model adaptively increases Δ to capture abrupt state changes (frame 15) and decreases it during stable period…

**Figure 8.** Figure 8: Performance comparison on the asynchronous subset [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: Zero-Shot Generalization on the AVSS dataset. The [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 36 canonical work pages

[1]

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexan- der Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. InEuropean Conference on Computer Vision. Springer, 213–229

work page 2020
[2]

Yaru Chen, Ruohao Guo, Xubo Liu, Peipei Wu, Guangyao Li, Zhenbo Li, and Wenwu Wang. 2023. CM-PIE: Cross-modal perception for interactive-enhanced audio-visual video parsing. arXiv:2310.07517 [cs.CV] https://arxiv.org/abs/2310. 07517

work page Pith review arXiv 2023
[3]

Yuanhong Chen, Yuyuan Liu, Hu Wang, Fengbei Liu, Chong Wang, Helen Frazer, and Gustavo Carneiro. 2024. Unraveling instance associations: A closer look for audio-visual segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26497–26507

work page 2024
[4]

Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. 2021. Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764(2021)

arXiv 2021
[5]

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. 2022. Masked-attention mask transformer for universal image seg- mentation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition. 1290–1299

work page 2022
[6]

Hahyeon Choi, Junhoo Lee, and Nojun Kwak. 2025. What’s Making That Sound Right Now? Video-centric Audio-Visual Localization. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 20095–20104

work page 2025
[7]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255

2009
[8]

Hao Fang, Tong Zhang, Xiaofei Zhou, and Xinxin Zhang. 2024. Learning Better Video Query with SAM for Video Instance Segmentation.IEEE Transactions on Circuits and Systems for Video Technology(2024)

work page 2024

Show all 50 references

[9]

Feng Gao, Xuepeng Jin, Xiaowei Zhou, Junyu Dong, and Qian Du. 2025. MSF- Mamba: Multiscale Feature Fusion State Space Model for Multisource Remote Sensing Image Classification.IEEE Transactions on Geoscience and Remote Sensing 63 (2025), 1–16. doi:10.1109/TGRS.2025.3535622

2025
[10]

Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. 2024. Avseg- former: Audio-visual segmentation with transformer. InProceedings of the AAAI conference on artificial intelligence, Vol. 38. 12155–12163

2024
[11]

Sitong Gong, Yunzhi Zhuge, Lu Zhang, Yifan Wang, Pingping Zhang, Lijun Wang, and Huchuan Lu. 2025. AVS-Mamba: Exploring Temporal and Multi- modal Mamba for Audio-Visual Segmentation.IEEE Transactions on Multimedia (2025), 1–13. doi:10.1109/TMM.2025.3542995

2025
[12]

Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752(2023)

2023 arXiv
[13]

Albert Gu, Karan Goel, and Christopher Ré. 2021. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396(2021)

2021 arXiv
[14]

Ruohao Guo, Xianghua Ying, Yaru Chen, Dantong Niu, Guangyao Li, Liao Qu, Yanyu Qi, Jinxing Zhou, Bowei Xing, Wenzhen Yue, et al. 2025. Audio-visual instance segmentation. InProceedings of the Computer Vision and Pattern Recog- nition Conference. 13550–13560

2025
[15]

Dawei Hao, Yuxin Mao, Bowen He, Xiaodong Han, Yuchao Dai, and Yiran Zhong
[16]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition. 770–778

2016
[17]

Miran Heo, Sukjun Hwang, Seoung Wug Oh, Joon-Young Lee, and Seon Joo Kim
[18]

Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, s...

2017
[19]

Shaofei Huang, Han Li, Yuqing Wang, Hongji Zhu, Jiao Dai, Jizhong Han, Wenge Rong, and Si Liu. 2023. Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation. arXiv:2309.09501 [cs.CV] https://arxiv.org/abs/2309.09501

2023 arXiv
[20]

Shaofei Huang, Rui Ling, Tianrui Hui, Hongyu Li, Xu Zhou, Shifeng Zhang, Si Liu, Richang Hong, and Meng Wang. 2025. Revisiting Audio-Visual Segmentation with Vision-Centric Transformer. InProceedings of the Computer Vision and Pattern Recognition Conference. 8352–8361

2025
[21]

Sukjun Hwang, Miran Heo, Seoung Wug Oh, and Seon Joo Kim. 2021. Video instance segmentation using inter-frame communication transformers.Advances in Neural Information Processing Systems34 (2021), 13352–13363

2021
[22]

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al
[23]

Harold W Kuhn. 1955. The Hungarian method for the assignment problem.Naval Research Logistics Quarterly2, 1-2 (1955), 83–97

1955
[24]

Jiaxu Li, Songsong Yu, Yifan Wang, Lijun Wang, and Huchuan Lu. 2024. SelM: Selective Mechanism based Audio-Visual Segmentation. InProceedings of the 32nd ACM International Conference on Multimedia(Melbourne VIC, Australia) (MM ’24). Association for Computing Machinery, New Yor...

2024
[25]

Jia Li, Yinfeng Yu, Liejun Wang, Fuchun Sun, and Wendong Zheng. 2025. Audio- Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation. arXiv:2509.16924 [cs.AI] https://arxiv.org/abs/2509.16924

2025
[26]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. InComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Procee...

2014
[27]

Leiye Liu, Miao Zhang, Jihao Yin, Tingwei Liu, Wei Ji, Yongri Piao, and Huchuan Lu. 2025. Defmamba: Deformable visual state space model. InProceedings of the Computer Vision and Pattern Recognition Conference. 8838–8847

2025
[28]

Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. 2024. Vmamba: Visual state space model. Advances in neural information processing systems37 (2024), 103031–103063

2024
[29]

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer us- ing shifted windows. InProceedings of the IEEE/CVF international conference on computer vision. 10012–10022

2021
[30]

Ilya Loshchilov and Frank Hutter. 2017. Decoupled Weight Decay Regularization. InInternational Conference on Learning Representations

2017
[31]

Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe. 2021. Hota: A higher order metric for evaluating multi-object tracking.International Journal of Computer Vision129 (2021), 548–578

2021
[32]

Luca Macesanu, Boueny Folefack, Samik Singh, Ruchira Ray, Ben Abbatematteo, and Roberto Martín-Martín. 2025. CAVER: Curious Audiovisual Exploring Robot. arXiv:2511.07619 [cs.RO] https://arxiv.org/abs/2511.07619

2025
[33]

Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Wei Ji, Jingjing Li, Yongri Piao, and Huchuan Lu. 2026. Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation. arXiv:2603.14203 [cs.CV] https://arxiv.org/abs/2603.14203

2026
[34]

Jinbae Seo, Hyeongjun Kwon, Kwonyoung Kim, Jiyoung Lee, and Kwanghoon Sohn. 2025. Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentation. arXiv:2509.22740 [eess.AS] https: //arxiv.org/abs/2509.22740

2025
[35]

Yunzhe Shen, Kai Peng, Leiye Liu, Wei Ji, Jingjing Li, Miao Zhang, Yongri Piao, and Huchuan Lu. 2025. Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation. arXiv:2509.18912 [cs.CV] https://arxiv. org/abs/2509.18912

2025
[36]

Weiss, Kevin Wilson, and John R

Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron J. Weiss, Kevin Wilson, and John R. Hershey. 2020. Unsupervised Sound Separation Using Mixture Invariant Training. arXiv:2006.12701 [eess.AS] https://arxiv.org/abs/2006.12701

2020 arXiv
[37]

Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, and Xiang Bai. 2022. Seqformer: Sequential transformer for video instance segmentation. InEuropean Conference on Computer Vision. Springer, 553–569

2022
[38]

Lanhu Wu, Zilin Gao, Hao Fei, Mong-Li Lee, and Wynne Hsu. 2025. LEAF-Mamba: Local Emphatic and Adaptive Fusion State Space Model for RGB-D Salient Object Detection. InProceedings of the 33rd ACM International Conference on Multimedia (Dublin, Ireland)(MM ’25). Association for ...

2025
[39]

Linjie Yang, Yuchen Fan, and Ning Xu. 2019. Video instance segmentation. In IEEE/CVF International Conference on Computer Vision. 5188–5197

2019
[40]

Qi Yang, Xing Nie, Tong Li, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, and Shiming Xiang. 2024. Cooperation does matter: Exploring multi-order bilateral relations for audio-visual segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

2024
[41]

Shusheng Yang, Xinggang Wang, Yu Li, Yuxin Fang, Jiemin Fang, Wenyu Liu, Xun Zhao, and Ying Shan. 2022. Temporally efficient vision transformer for video instance segmentation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition. 2885–2895

2022
[42]

Tao Zhang, Xingye Tian, Yu Wu, Shunping Ji, Xuebo Wang, Yuan Zhang, and Pengfei Wan. 2023. Dvis: Decoupled video instance segmentation framework. In IEEE/CVF International Conference on Computer Vision. 1282–1291

2023
[43]

Jialong Zhong, Tingwei Liu, Baokun Yue, Jingjing Li, Yongri Piao, Miao Zhang, Leiye Liu, Jiahong Jiang, Wei Ji, and Huchuan Lu. 2026. AdaSurvMamba: Dynamic MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Leiye Liu et al. Fusion and Semantic Scanning for Multimodal Surviva...

2026 arXiv
[44]

Jinxing Zhou, Dan Guo, Ruohao Guo, Yuxin Mao, Jingjing Hu, Yiran Zhong, Xiaojun Chang, and Meng Wang. 2025. Towards Open-Vocabulary Audio-Visual Event Localization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8362–8371

2025
[45]

Jinxing Zhou, Dan Guo, and Meng Wang. 2023. Contrastive Positive Sample Propagation Along the Audio-Visual Event Line.IEEE Transactions on Pattern Analysis and Machine Intelligence45, 6 (2023), 7239–7257. doi:10.1109/TPAMI. 2022.3223688

2023
[46]

Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. 2022. Audio– Visual Segmentation. InComputer Vision – ECCV 2022, Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and...

2022
[47]

Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. 2024. Vision mamba: efficient visual representation learning with bidirectional state space model. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’...

2024
[2022]

Vita: Video instance segmentation via object token association.Advances in Neural Information Processing Systems35 (2022), 23109–23120

2022
[2023]

InIEEE/CVF International Conference on Computer Vision

Segment anything. InIEEE/CVF International Conference on Computer Vision. 4015–4026
[2024]

In Proceedings of the AAAI conference on artificial intelligence, Vol

Improving audio-visual segmentation with bidirectional generation. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38. 2067–2075

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexan- der Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. InEuropean Conference on Computer Vision. Springer, 213–229

work page 2020

[2] [2]

Yaru Chen, Ruohao Guo, Xubo Liu, Peipei Wu, Guangyao Li, Zhenbo Li, and Wenwu Wang. 2023. CM-PIE: Cross-modal perception for interactive-enhanced audio-visual video parsing. arXiv:2310.07517 [cs.CV] https://arxiv.org/abs/2310. 07517

work page Pith review arXiv 2023

[3] [3]

Yuanhong Chen, Yuyuan Liu, Hu Wang, Fengbei Liu, Chong Wang, Helen Frazer, and Gustavo Carneiro. 2024. Unraveling instance associations: A closer look for audio-visual segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26497–26507

work page 2024

[4] [4]

Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. 2021. Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764(2021)

arXiv 2021

[5] [5]

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. 2022. Masked-attention mask transformer for universal image seg- mentation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition. 1290–1299

work page 2022

[6] [6]

Hahyeon Choi, Junhoo Lee, and Nojun Kwak. 2025. What’s Making That Sound Right Now? Video-centric Audio-Visual Localization. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 20095–20104

work page 2025

[7] [7]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255

2009

[8] [8]

Hao Fang, Tong Zhang, Xiaofei Zhou, and Xinxin Zhang. 2024. Learning Better Video Query with SAM for Video Instance Segmentation.IEEE Transactions on Circuits and Systems for Video Technology(2024)

work page 2024