Pith. sign in

REVIEW 3 major objections 6 minor 47 references

SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAM2-LOVE compresses text, audio, and vision into a single learnable token that prompts SAM2 to segment referred objects in videos, beating prior methods by 8.5 J&F points on Ref-AVS.

desk verdict Solid engineering with a big number on a niche benchmark; the video-level gain needs a stronger causal control. read the letter →

arxiv 2506.01558 v1 pith:BO35SUXE submitted 2025-06-02 cs.CV

Yuji Wang , Haoran Xu , Yong Liu , Jiaze Li , Yansong Tang This is my paper
classification cs.CV
keywords ReferenceAudio-VisualSegmentationSAM2multimodalfusionlearnabletokenvideoobjectpropagationaccumulationRef-AVS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that a pretrained video segmentation model can be extended to three-modality referring segmentation without retraining its tracking machinery. On the Ref-AVS benchmark it reports 58.5 J&F, beating the previous best method by 8.5 points, and a 12.2-point gain on object categories never seen in training. The proposed trick is to compress text, audio, and visual features into one learnable token that is fed to SAM2's prompt encoder on the first frame, letting SAM2's built-in memory attention propagate the mask through the video. The paper argues that this works because the fusion module gives the token the needed semantic and positional cues, while token propagation and accumulation keep the prompt from forgetting earlier frames. If correct, the result shows that adding new input modalities to a frozen video segmenter can be cheap: only the fusion module and prompt-related parts are trained.

What carries the argument

The central object is the learnable [seg] token, a single vector that is prepended to the sequence of projected audio, text, and visual embeddings and refined by a bidirectional fusion transformer. Token propagation feeds the output [seg] of one frame into the next so that the prompt becomes a running summary that mixes spatial and temporal information; token accumulation appends the global [cls] representations of all previous frames to the current input, acting as a backward knowledge flow that prevents early frames from being forgotten. The [seg] token is then projected and passed to SAM2's prompt encoder, whose mask decoder and memory attention are fine-tuned on the first frame only, so that SAM2's pretrained video tracking carries the segmentation across all frames.

What would settle it

Construct videos where the referred object is present only after the first frame (for instance, it enters at frame 10). If the model is prompted on frame 1 and the mask does not appear when the object enters, the assumption that a first-frame-based token carries enough object identity breaks. A second check: hold text and audio fixed while changing the target's appearance gradually; if J&F drops sharply after a few frames, the token is not preserving the object's identity through propagation.

Watch

Extended reading notes

Core claim

On the paper's own terms, SAM2-LOVE is the first framework to apply SAM2 to Language-aided Audio-Visual Scenes, and it claims that a single [seg] token can carry all three modalities into the model's prompt encoder. The token is produced by a six-layer fusion transformer that receives the projected audio, text, and visual embeddings of the current frame, plus the [seg] token from the previous frame and the accumulated [cls] history tokens of all earlier frames. Training supervises only the first frame with a BCE-plus-DICE loss, and the same [seg] token prompts SAM2 during inference-video tracking. The reported result is 58.5 J&F on the Mix split, 8.5 points above the prior SOTA, with a particularly large 12.2-point gain on unseen categories.

Load-bearing premise

The load-bearing premise is that a single token assembled from the first frame's visual features plus the full text and audio can be read by SAM2's prompt encoder and direct the frozen memory-attention modules to keep tracking the right object even when it changes appearance, leaves the view, or re-enters.

Editorial extensions

If this is right

  • On Ref-AVS, the reported 58.5 J&F beats the previous best, EEMC, by 8.5 points, and the unseen-category gap is 12.2 points (69.4 vs. 57.2).
  • Because SAM2's image encoder and memory attention stay frozen, adapting the model to text-plus-audio referring segmentation requires training only the fusion transformer, the [seg] token, and SAM2's prompt encoder and mask decoder.
  • The null-reference scores show the method's known weakness: with an empty or invalid reference, SAM2 tends to segment some region anyway, giving an S score of 0.23 versus EEMC's 0.007.
  • Token accumulation adds historical frame representations at no extra training cost and, per the ablation, improves performance on both seen and unseen splits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to swap the fusion module's input encoders for other modalities (depth, heat, or object-detection cues) and test whether the same frozen-SAM2 recipe transfers; the paper's design suggests the fusion module is modality-agnostic, but that claim is not tested here.
  • The large unseen-category gain may be partially an artifact of EEMC overfitting to seen categories rather than a pure advantage of SAM2's tracking; a fair comparison on categories that appear in both training distributions would isolate the effect.
  • The token propagation and accumulation scheme could be viewed as a lightweight memory bank; an ablation that replaces [cls] accumulation with simple feature concatenation could reveal whether the backward flow is necessary or just a form of feature reuse.
  • A confidence gate on the [seg] token before prompting could address the null-reference failure, since the failure mode is over-segmentation of irrelevant regions when no referent exists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents SAM2-LOVE, a framework for Reference Audio-Visual Segmentation (Ref-AVS) that fuses text, audio, and visual representations into a learnable [seg] token via a transformer encoder. The method uses token propagation across frames and [cls] token accumulation to preserve historical information, and the resulting token prompts SAM2 on the first frame so that SAM2's memory attention can propagate masks through the video. Training supervises only the first frame while the fusion module has full video access. On the Ref-AVS benchmark, the method reports 58.5 J&F, outperforming the prior SOTA EEMC by 8.5 points, with a 12.2-point gain on unseen categories. Ablations study backbone choices, fusion transformer depth, number of seg tokens, fusion strategy, and trainable SAM2 modules.

Significance. If the reported result is reproducible, the paper offers a conceptually simple recipe: a frozen pretrained video segmentation model can be adapted to three-modality referring segmentation by training only a small fusion module and the SAM2 prompt decoder. The method is clearly specified, the code is promised, and the ablations are informative and largely consistent with the design choices. The principal weakness is that the central causal attribution—that SAM2's memory tracking drives the video-level gains—is underdetermined because the fusion module has full video access at inference and no first-frame-only control is reported. The lack of variance or significance testing also leaves the magnitude of the headline improvement uncertain. The paper is a reasonable empirical contribution, but additional controls are needed to support the claimed mechanism.

major comments (3)
  1. [Section 3.3, Figure 3] The paper states that the fusion transformer and [seg] token are trained with full access to the entire video sequence and that during testing both the fusion module and SAM2 receive the entire video. If the [seg] token used to prompt SAM2 on the first frame is the output after propagation through all frames, then the prompt is an offline video summary that can encode where the target appears in later frames. The reported 58.5 J&F (Table 1) could then be driven by the fusion module's future-frame information rather than by SAM2's zero-shot memory tracking. The paper does not report an ablation that restricts the fusion module to the first frame at inference, nor one that disables or isolates SAM2's memory attention. Please add these causal controls, and if the prompt does not in fact use future-frame information, specify the inference data flow precisely enough to rule out this concern.
  2. [Tables 1-6] All quantitative results, including the headline 8.5-point J&F improvement over EEMC, are reported from a single run without variance or significance testing. Several ablation comparisons are small (e.g., Table 3: 1-layer versus 6-layer fusion transformer differs by 1.5 points on unseen J&F; Table 4: 1 versus 8 [seg] tokens differs by 0.6 points on unseen J&F). Without multiple seeds and an appropriate paired significance test, the statistical reliability of the main claim and of these design decisions is unknown. Please report mean and standard deviation over at least three seeds and include a significance test.
  3. [Tables 2-6] The ablations vary backbones, fusion depth, token count, fusion strategy, and trainable SAM2 modules, but none removes one of the three modality branches. The paper's central conceptual claim is that all three modalities, fused into a single token, are necessary for the Ref-AVS gains. A control that drops the audio branch or the text branch would directly evidence that attribution. Without such a control, the gains could in principle be driven by a two-modality subset alone, and the contribution of the third modality remains unquantified.
minor comments (6)
  1. [Figure 3 caption] There is a typo: 'pipline' should be 'pipeline'.
  2. [Section 3.2, Eq. (3)] The placement of the [aud] and [vis] tokens is confusing. The text says these tokens indicate the end and beginning of their corresponding sequences, but the ordering in Eq. (3) places [aud] after the audio embeddings and [vis] before the visual embeddings, with no explicit token for the text sequence. Please clarify the intended role and ordering with an annotated example.
  3. [Section 4.5] The NULL split discussion is hard to follow: the paper reports an S score of 0.23 for SAM2-LOVE versus 0.007 for EEMC, and then says 0.23 is 'similar results with the AVSBench plus text method' (which has 0.208). The direction of the metric and what constitutes better performance should be stated explicitly, and the comparison to AVSBench+text should be rephrased.
  4. [Figure 5 caption] The phrase 'we use the legends of the trumpet to represent the different sound volumes' is unclear. Please rephrase to explain the visual encoding of sound volume.
  5. [References] References [28] and [30] are the same work (GAVS) cited in two different venues. Please unify them to avoid duplicate citations.
  6. [Table 6] The ablation on trainable SAM2 modules is reported only on the Seen split. Reporting the Unseen split would make the comparison complete and more informative.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark result is an external empirical comparison, not a derivation from the paper's own outputs.

full rationale

SAM2-LOVE's central claim is an empirical performance comparison on the public Ref-AVS benchmark against external baselines, using externally defined J&F metrics computed from ground-truth masks. The pretrained encoders (VGGish, ViT, DistilRoBERTa) and SAM2 weights are external resources, and no fitted parameter is used to define the target metric. The fusion transformer and [seg] token are trainable components, but their outputs are evaluated against ground truth, and the ablations in Tables 2-6 test design choices rather than redefining the evaluation. The paper's self-citations (e.g., [1], [17], [18], [20], [33], [40], [41]) are related-work context, not load-bearing justification of the main result. The limitation section honestly reports weak NULL-rejection performance, which is a substantive weakness rather than a circular move. The skeptic's concern that the fusion module sees the full video during training and inference (Section 3.3) is a causal-attribution question about whether the gain comes from SAM2's memory tracking or from full-video fusion; it is underdetermination and a missing-control issue, not equivalence-by-construction, so it does not constitute circularity under the specified definitions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central contribution is empirical: no new physics or mathematics is derived. It rests on pretrained encoder transfer (VGGish, ViT, DistilRoBERTa), on SAM2's ability to be steered by a learned token, and on the correctness of the Ref-AVS benchmark. The only hand-chosen values are architecture and training hyperparameters selected by ablation on the same benchmark. No new physical entities are introduced.

free parameters (3)
  • Fusion transformer depth = 6 layers
    Chosen by ablation in Table 3; 12 layers give slightly better unseen performance but 6 is used for efficiency. This is a hyperparameter selection, not a physical constant.
  • Number of [seg] tokens = 1
    Chosen by ablation in Table 4; 8 tokens improve the unseen split but hurt the seen split, so 1 is used by default.
  • Loss weights lambda_bce and lambda_dice = 1.0 each
    Set by hand in Section 4.1 without reported sensitivity analysis; standard weights for a combined BCE and Dice mask loss.
assumptions (4)
  • domain assumption Pretrained VGGish, ViT, and DistilRoBERTa embeddings transfer to the Ref-AVS domain without fine-tuning.
    Section 3.2: encoders are frozen and their outputs are projected with MLPs. If these representations do not align with LAVS semantics, the fusion token cannot encode the target.
  • domain assumption SAM2's mask decoder and memory propagation can be driven by a single learned prompt token projected into its prompt embedding space.
    Section 3.3 and Figure 3: only the prompt encoder and mask decoder are trained; image encoder and memory attention are frozen. The whole method depends on this transfer.
  • domain assumption Ref-AVS benchmark masks and splits are correct and the reported metric is comparable across methods.
    Section 4.1: performance is judged on the Ref-AVS test set. Any label noise or split inconsistency would propagate to the central claim.
  • standard math Standard transformer self-attention can compress three modalities into one token that retains enough spatial and temporal information.
    Equations (2)-(4): the fusion transformer is a standard bidirectional encoder; the paper does not prove a representational guarantee and relies on empirical validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes." pith.science (2026). https://pith.science/paper/BO35SUXE

@misc{pith2026250601558,
  author       = {Pith},
  title        = {Pith review of: SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BO35SUXE}},
  note         = {Machine review of arXiv:2506.01558}
}
abstract

Reference Audio-Visual Segmentation (Ref-AVS) aims to provide a pixel-wise scene understanding in Language-aided Audio-Visual Scenes (LAVS). This task requires the model to continuously segment objects referred to by text and audio from a video. Previous dual-modality methods always fail due to the lack of a third modality and the existing triple-modality method struggles with spatio-temporal consistency, leading to the target shift of different frames. In this work, we introduce a novel framework, termed SAM2-LOVE, which integrates textual, audio, and visual representations into a learnable token to prompt and align SAM2 for achieving Ref-AVS in the LAVS. Technically, our approach includes a multimodal fusion module aimed at improving multimodal understanding of SAM2, as well as token propagation and accumulation strategies designed to enhance spatio-temporal consistency without forgetting historical information. We conducted extensive experiments to demonstrate that SAM2-LOVE outperforms the SOTA by 8.5\% in $\mathcal{J\&F}$ on the Ref-AVS benchmark and showcase the simplicity and effectiveness of the components. Our code will be available here.

Figures

Figures reproduced from arXiv: 2506.01558 by the authors.

Figure 1
Figure 1. The comparison with dual-modality SAM pipelines. (a) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the Ref-AVS task under varying modality cues. The expression can be treated as a kind of control signal to [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The pipline of SAM2-LOVE. We harness a fusion transformer to compress the tripe modality representations into a learnable [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The token propagation and token accumulation strategy in the fusion transformer. For the token propagation, the output [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The visualization results of the referred objects in the Ref-AVS. Notably, we use the legends of the trumpet to represent the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 24 canonical work pages

  1. [1]

    Self-calibrated clip for training-free open-vocabulary segmentation

    Sule Bai, Yong Liu, Yifei Han, Haoji Zhang, and Yansong Tang. Self-calibrated clip for training-free open-vocabulary segmentation. arXiv preprint arXiv:2411.15869, 2024. 2

    arXiv 2024

  2. [2]

    Unraveling in- stance associations: A closer look for audio-visual segmenta- tion

    Yuanhong Chen, Yuyuan Liu, Hu Wang, Fengbei Liu, Chong Wang, Helen Frazer, and Gustavo Carneiro. Unraveling in- stance associations: A closer look for audio-visual segmenta- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 26497–26507,

  3. [3]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3

    arXiv 2010

  4. [4]

    Avsegformer: Audio-visual segmentation with trans- former

    Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio-visual segmentation with trans- former. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12155–12163, 2024. 3, 6

    work page 2024

  5. [5]

    Audio set: An ontology and human- labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In 2017 IEEE interna- tional conference on acoustics, speech and signal process- ing, pages 776–780. IEEE, 2017. 3

    work page 2017

  6. [6]

    Cnn archi- tectures for large-scale audio classification

    Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn archi- tectures for large-scale audio classification. In2017 IEEE in- ternational conference on acoustics, speech and signal pro- cessing, pages 131–135. IEEE, 2017. 3

    work page 2017

  7. [7]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 con- ference on empirical methods in natural language process- ing, pages 787–798, 2014. 2

    work page 2014

  8. [8]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 2, 3

    2023

Show all 47 references
  1. [9]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9579–9589, 2024. 2

    2024

  2. [10]

    Learning to answer questions in dynamic audio-visual scenarios

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji- Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108–19118, 2022. 1, 3

    2022

  3. [11]

    Robust referring video object segmentation with cyclic structural consensus

    Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj, and Yan Lu. Robust referring video object segmentation with cyclic structural consensus. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22236– 22245, 2023. 6

    2023

  4. [12]

    Libero: Benchmarking knowl- edge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024. 8

    2024

  5. [13]

    GRES: Gen- eralized referring expression segmentation

    Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Gen- eralized referring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2023. 2

    2023

  6. [14]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2

    2024

  7. [15]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  8. [16]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364, 2019. 3

    1907 arXiv

  9. [17]

    Open-vocabulary segmentation with semantic-assisted calibration

    Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3491– 3500, 2024. 2

    2024

  10. [18]

    Universal segmentation at arbi- trary granularity with language instruction

    Yong Liu, Cairong Zhang, Yitong Wang, Jiahao Wang, Yujiu Yang, and Yansong Tang. Universal segmentation at arbi- trary granularity with language instruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 3459–3469, 2024. 2

    2024

  11. [19]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 6

    2019

  12. [20]

    Soc: Semantic-assisted object cluster for referring video object segmentation

    Zhuoyan Luo, Yicheng Xiao, Yong Liu, Shuyan Li, Yi- tong Wang, Yansong Tang, Xiu Li, and Yujiu Yang. Soc: Semantic-assisted object cluster for referring video object segmentation. Advances in Neural Information Processing Systems, 36, 2024. 6

    2024

  13. [21]

    Mod- eling context between objects for referring expression un- derstanding

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Mod- eling context between objects for referring expression un- derstanding. In IEEE European Conference on Computer Vision, pages 792–807. Springer, 2016. 2

    2016

  14. [22]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

    2024 arXiv

  15. [23]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374–26383, 2024. 2

    2024

  16. [24]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    1910 arXiv

  17. [25]

    Efficient attention: Attention with lin- ear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with lin- ear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531– 3539, 2021. 8

    2021

  18. [26]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  19. [27]

    Auto- acd: A large-scale dataset for audio-language representation learning

    Luoyi Sun, Xuenan Xu, Mengyue Wu, and Weidi Xie. Auto- acd: A large-scale dataset for audio-language representation learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5025–5034, 2024. 3

    2024

  20. [28]

    Prompting segmentation with sound is generalizable audio-visual source localizer

    Yaoting Wang, Weisong Liu, Guangyao Li, Jian Ding, Di Hu, and Xi Li. Prompting segmentation with sound is generalizable audio-visual source localizer. arXiv preprint arXiv:2309.07929, 2023. 6

    2023 arXiv

  21. [29]

    Efficient remote sensing transformer for coastline detection with sentinel-2 satellite imagery

    Yuji Wang, Ruojun Zhao, and Zijun Sun. Efficient remote sensing transformer for coastline detection with sentinel-2 satellite imagery. In IGARSS 2023-2023 IEEE International Geoscience and Remote Sensing Symposium , pages 5439–

    2023

  22. [30]

    Prompting segmentation with sound is gen- eralizable audio-visual source localizer

    Yaoting Wang, Weisong Liu, Guangyao Li, Jian Ding, Di Hu, and Xi Li. Prompting segmentation with sound is gen- eralizable audio-visual source localizer. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5669– 5677, 2024. 2, 3

    2024

  23. [31]

    Ref-avs: Refer and segment objects in audio-visual scenes

    Yaoting Wang, Peiwen Sun, Dongzhan Zhou, Guangyao Li, Honggang Zhang, and Di Hu. Ref-avs: Refer and segment objects in audio-visual scenes. IEEE European Conference on Computer Vision, 2024. 1, 2, 3, 5, 6

    2024

  24. [32]

    Convolution meets trans- former: Efficient hybrid transformer for semantic segmenta- tion with very high resolution imagery

    Yuji Wang, Ruojun Zhao, Shicai Wei, Jingchen Ni, Meng Wu, Yang Luo, and Chunbo Luo. Convolution meets trans- former: Efficient hybrid transformer for semantic segmenta- tion with very high resolution imagery. In IGARSS 2024- 2024 IEEE International Geoscience and Remote Sensin...

    2024

  25. [33]

    Iterprime: Zero-shot referring image segmen- tation with iterative grad-cam refinement and primary word emphasis

    Yuji Wang, Jingchen Ni, Yong Liu, Chun Yuan, and Yan- song Tang. Iterprime: Zero-shot referring image segmen- tation with iterative grad-cam refinement and primary word emphasis. arXiv preprint arXiv:2503.00936, 2025. 2

    2025 arXiv

  26. [34]

    Onlinerefer: A simple online baseline for referring video object segmentation

    Dongming Wu, Tiancai Wang, Yuang Zhang, Xiangyu Zhang, and Jianbing Shen. Onlinerefer: A simple online baseline for referring video object segmentation. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 2761–2770, 2023. 2

    2023

  27. [35]

    Language as queries for referring video object seg- mentation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4974– 4984, 2022. 6

    2022

  28. [36]

    Language as queries for referring video object seg- mentation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4974– 4984, 2022. 2

    2022

  29. [37]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858–3869, 2024. 2

    2024

  30. [38]

    Efficientsam: Leveraged masked image pretraining for efficient segment anything

    Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xi- ang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. In Proceedings of the IEEE/CVF Conference on Compu...

    2024

  31. [39]

    Avqa: A dataset for audio- visual question answering on videos

    Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio- visual question answering on videos. In Proceedings of the 30th ACM International Conference on Multimedia , pages 3480–3491, 2022. 1, 3

    2022

  32. [40]

    Lavt: Language-aware vision transformer for referring image segmentation

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Heng- shuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2

    2022

  33. [41]

    Language- aware vision transformer for referring segmentation

    Zhao Yang, Jiaqi Wang, Xubing Ye, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Language- aware vision transformer for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence,

  34. [42]

    Faster segment anything: Towards lightweight sam for mo- bile applications

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mo- bile applications. arXiv preprint arXiv:2306.14289 , 2023. 3

    2023 arXiv

  35. [43]

    Evf-sam: Early vision-language fusion for text-prompted segment anything model

    Yuxuan Zhang, Tianheng Cheng, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xing- gang Wang. Evf-sam: Early vision-language fusion for text-prompted segment anything model. arXiv preprint arXiv:2406.20076, 2024. 2, 3

    2024 arXiv

  36. [44]

    Fast segment any- thing

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment any- thing. arXiv preprint arXiv:2306.12156, 2023. 3

    2023 arXiv

  37. [45]

    Audio-visual segmentation

    Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In European Conference on Computer Vision, 2022. 3, 6

    2022

  38. [46]

    Audio- visual segmentation with semantics

    Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Ling- peng Kong, Meng Wang, and Yiran Zhong. Audio- visual segmentation with semantics. arXiv preprint arXiv:2301.13190, 2023. 3

    2023 arXiv

  39. [47]

    Generalized decoding for pixel, image, and lan- guage

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and lan- guage. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.