Pith. sign in

REVIEW 2 major objections 3 minor 71 references

The paper claims a frozen vision-language embedding base can be extended to audio without touching a single base weight, using a small connector and modality-gated adapters, so audio joins text, image, and video in one space while base outp

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:41 UTC pith:KI7HKTPP

load-bearing objection Core recipe is sound and honestly reported; the 'emergent' audio–image claim needs a VGGSound-contamination check before it's taken at face value. the 2 major comments →

arxiv 2607.18666 v1 pith:KI7HKTPP submitted 2026-07-21 cs.CL cs.SD

Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

Abdul Basit Tonmoy , Kazi Fardinul Hoque , Md. Shahrier Islam Arham , Arman Luthra This is my paper
classification cs.CL cs.SD
keywords multimodal embedding spaceaudio-text retrievalfrozen backbone adaptationemergent cross-modal retrievalmodality-gated adapterscontrastive learningzero-shot transferretrieval invariance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a single embedding space for text, image, video, and audio can be built without retraining the underlying vision-language model. It claims that training only a small connector between a frozen audio encoder and a frozen vision-language embedding base, aligning audio to text alone, makes audio a first-class modality, and that audio-to-image retrieval emerges as a side effect because the base already binds text to images. A second generation shows where the remaining gap lives: inside the frozen decoder layers, so it adds per-layer adapters gated to fire only on audio inputs, keeping text, image, and video outputs bit-for-bit identical to the released base. A sympathetic reader should care because the recipe preserves already-deployed embeddings and benchmark results by construction, and it maps the design space with controlled negative results rather than only positive numbers.

Core claim

The central claim is that a frozen vision-language embedding model can be extended with audio without updating any of its weights. The paper demonstrates this with two generations: a 16.4M-parameter connector trained contrastively against cached frozen-text embeddings already reaches strong audio-to-text retrieval and produces emergent audio-to-image retrieval at 29 times chance despite zero audio-visual training pairs; a second generation adds 44.2M parameters of per-layer adapters, gated so they execute only for audio, improving text-to-audio retrieval while text, image, and video outputs remain bit-for-bit identical to the released base. The paper locates the remaining bottleneck in the f

What carries the argument

The load-bearing mechanism is the text pivot: audio is aligned only to text, and audio-to-image and audio-to-video retrieval inherit the base's existing text-image geometry. Two trained components carry this. Generation 1 uses a perceiver-style resampler connector (about 16.4M parameters), a small attention module that compresses audio frames into a fixed number of tokens injected at placeholder positions in the frozen decoder. Generation 2 keeps the connector and adds a bottleneck adapter to the residual stream of every decoder layer, hard-gated so the adapter runs only while audio is being encoded. The gate converts 'base outputs are preserved' from an empirical hope into a construction-ti

Load-bearing premise

The base model's existing text-image alignment is a trustworthy bridge for the caption register used in training and evaluation, and the paper does not independently measure or control that geometry.

What would settle it

Train the same connector and adapters against the same audio-caption data but attach them to a vision-language base whose text-image alignment has been deliberately weakened or scrambled, for example by randomizing or freezing the text projector on sound-event captions. If emergent audio-to-image retrieval stays well above chance, the 'emergent' claim is not carrying the result; if it collapses to chance while audio-to-text retrieval stays intact, the paper's mechanism is confirmed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Text, image, and video embeddings produced by the base remain valid bit-for-bit, so an existing index never needs re-embedding when audio is added.
  • Audio-to-text retrieval reaches roughly 0.74 recall at rank 10 on a standard sound-caption benchmark with both towers frozen, and transfers zero-shot to a held-out caption benchmark.
  • Audio-to-image retrieval emerges at about 29 times chance with no audio-visual supervision, and improves in lockstep with audio-to-text quality.
  • The negative results redraw the design space: cleaner rewritten captions, a leaderboard-stronger audio tower, and a wider connector all reduce retrieval, while gated in-layer capacity helps.
  • Both generations train in hours on a single accelerator, making the recipe reproducible without a large compute budget.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the large protocol effect implies that published comparisons among LLM-backbone embedding models are only meaningful when the input template is matched; many reported gaps may be partly formatting artifacts.
  • Editorial inference: the negative results suggest scaling this recipe is not primarily a data problem; the next testable lever is matching the caption register of the target benchmark rather than making captions cleaner.
  • Editorial inference: the modality-gated adapter pattern could transfer to any future modality, such as depth, sensor streams, or new audio formats, with the same bitwise-preservation guarantee, and that extension is directly testable.
  • Editorial inference: the full-corpus negative bank ceiling near half a million pairs implies that the scaling path requires improved negative sampling, not just more data; soft-label and false-negative masking terms are a first step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper presents two generations of an audio extension to a frozen vision-language embedding model. Generation 1 trains only a 16.4M-parameter perceiver-resampler connector between a frozen Qwen2.5-Omni audio tower and a byte-frozen Qwen3-VL-Embedding-2B base; generation 2 adds 44.2M modality-gated bottleneck adapters at every decoder layer, hard-gated so non-audio inputs are bitwise identical to the released base. Training uses a symmetric InfoNCE objective at all Matryoshka rungs with a full-corpus frozen-text negative bank, native chat-template targets, diagonal whitening, and an in-domain fine-tuning stage. The paper reports AudioCaps audio-to-text R@10 of 0.741/0.743, strictly zero-shot Clotho transfer, emergent audio-to-image retrieval on VGGSound-696 at R@10 0.418/0.392 (29× chance), three controlled negative results, and a training-protocol effect of +14.5 R@10 points. Weights, code, and evaluation harnesses are released.

Significance. If the results hold, this is a practically valuable recipe: it adds audio to a deployed frozen embedding stack at small training cost while preserving, by construction and by test, the base's text/image/video outputs bit-for-bit. The paper is unusually thorough: machine-checked invariance tests, a loss-floor audit, controlled A/B experiments, explicit single-seed noise caveats, and open weights/code are all real strengths. The central novelty, however, is the 'emergent' audio-to-image retrieval claim, and that claim currently lacks a critical control on the frozen base's own pretraining data. The audio-text retrieval results and the invariance guarantee stand independently; the emergence interpretation needs tightening before the paper's strongest conclusion is fully supported.

major comments (2)
  1. [§7, §6.1] The headline 'emergent audio→image' result has a load-bearing gap: the VGGSound blacklist applies to the connector/adapter training corpus, not to the pretraining data of the frozen base. If Qwen3-VL-Embedding-2B saw VGGSound frames/captions, the text→image leg is directly supervised for the exact evaluation clips, and 'zero paired audio-visual training data' is true only for the trained components. Please add an audit/control (e.g., evaluate on clips/captions that cannot be in the base's pretraining, or compare with a base ablated at the text-image leg) or explicitly reframe the claim as compositional transfer through a pretrained text→image space, not zero-shot cross-modal emergence.
  2. [§7] The stated mechanism — audio inherits text→image geometry via caption embeddings — is at odds with the reported numbers: audio→image R@10 is 0.418 for v0.2, while text↔image R@10 on the same VGGSound-696 protocol is 0.331/0.319. If audio embeddings are trained to sit near whitened text embeddings, the caption→image retrieval should upper-bound the audio→image result; the observed inversion needs a quantitative explanation (differences in text-query formatting, whitening statistics, or per-modality mean-centering between the text and audio query paths). Without this, the mechanism claim is not supported by the paper's own tables.
minor comments (3)
  1. [Abstract] Typo: 'widening the connector eachreduceretrieval' should read 'each reduce retrieval'.
  2. [Table 6 caption] The header 'Gen. 1 (v0.3) Gen. 2' is visually ambiguous; add column separators or vertical rules for clarity.
  3. [§6.1, Uncertainty] The binomial standard errors assume independent queries; VGGSound items are grouped by class and the caption pools overlap, so the effective noise may be larger than stated. This does not change the paper's qualitative conclusions but should be noted.

Circularity Check

0 steps flagged

No significant circularity: the audio-to-image result is an empirical composition of trained audio-text alignment with the frozen base's text-image geometry, and bitwise invariance is mechanically enforced and tested.

full rationale

The paper's derivation chain is not circular. The connector/adapters are trained only on audio-text pairs (Eq. 1, InfoNCE against frozen text targets), and VGGSound is blacklisted from that training; emergent audio-to-image retrieval is therefore an out-of-distribution empirical result, not a quantity fitted during training. The load-bearing premise that the base already places text near images is an external property of the frozen Qwen3-VL-Embedding-2B base, not an output of this paper's optimization, and the paper explicitly tests the transfer path (audio-to-text gains compound into audio-to-image gains while text-image cells remain identical). The bitwise-invariance claim is also non-circular: it is a mechanical consequence of freezing all base parameters and gating adapters off for non-audio inputs, and the paper verifies it with unit tests, drift assertions, and release smoke tests rather than assuming it. The Clotho numbers are strictly zero-shot by construction, and the paper includes controlled negative results and protocol-matched baselines. The skeptic concern that the frozen base's own pretraining may overlap VGGSound data is a possible data-contamination/benchmark-integrity issue, not a circular derivation; it does not make the audio-to-image result equivalent to the training objective. No load-bearing self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction were found. The acknowledged limitations (single cross-modal benchmark, mechanical rather than re-measured preservation, single-sourced baseline numbers) are measurement caveats, not circular steps.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 2 invented entities

The central claim rests on a small set of hand-chosen hyperparameters and architectural widths, plus domain assumptions about the frozen towers' learned semantics. No new physical entities are postulated; the new modules are open-source artifacts with testable behavior.

free parameters (7)
  • learnable temperature = initialized log(1/0.07), clamped at log(100); trained value not reported
    Scales InfoNCE logits; trained alongside the connector (§3.1, §4.5).
  • CORAL penalty λ = 0.05
    Hand-set weight on covariance penalty in Eq. (1); no ablation shown.
  • soft-label β = 0.3
    Enabled for 500K+ corpora; reported +2.6 R@10 in a matched A/B (§4.1, §5.4).
  • false-negative mask threshold τ = 0.98
    Chosen from caption-similarity analysis (§5.3); used to mask bank entries.
  • connector width d_r and latent count = 384 / 64 queries / 6 blocks
    Selected by A/B: d_r=512 memorizes, d_r=256 is worse (§8.3).
  • adapter rank = 384 (44.2M params)
    Rank-384 beats rank-128; had not saturated at the largest rank trained (§6.2, Appendix C).
  • per-dimension whitening statistics = mean/std fitted from cached frozen-text captions
    Diagonal whitening on the frozen text side, fitted once from caption embeddings (§4.3).
axioms (5)
  • domain assumption InfoNCE contrastive loss with symmetric audio-text directions aligns audio and text embeddings in a shared space.
    Core training objective, Eq. (1); assumes frozen text embeddings are a valid target distribution.
  • domain assumption The base model Qwen3-VL-Embedding-2B already places semantically related text, images, and video close together.
    Invoked in §7: 'captions of a barking dog live near images of barking dogs by the base’s own training'; the emergent audio-image claim depends on this inherited geometry.
  • domain assumption The frozen Qwen2.5-Omni audio tower's post-projection features are sufficient for retrieval after a learned connector.
    Used throughout; the tower-swap negative result (§8.2) shows not all audio towers work, so this assumption is load-bearing.
  • standard math Diagonal whitening preserves Matryoshka truncation commutativity.
    Stated in §4.3: diagonal transform with first-D statistics commutes with truncation; full covariance would not.
  • domain assumption Single-seed evaluations and binomial standard errors are adequate to support the paper's headline differences.
    §6.1 states all results are single-seed and deltas below ~2 points are treated as parity; this weakens some small-difference claims.
invented entities (2)
  • FusionResampler (perceiver-resampler connector) independent evidence
    purpose: Maps 3584-d audio-tower frames to 64 tokens in the base LLM's input embedding space.
    Released code and checkpoints; gradients/invariance are unit-tested (§3.1, Appendix F).
  • Modality-gated deep adapters independent evidence
    purpose: Add per-layer audio capacity inside the frozen decoder without executing on non-audio inputs, preserving bitwise outputs.
    Released code/checkpoints; bitwise invariance tests demonstrate the gate (§3.2, §6.4).

pith-pipeline@v1.3.0-alltime-deepseek · 26754 in / 14045 out tokens · 115706 ms · 2026-08-01T14:41:09.984370+00:00 · methodology

0 comments
read the original abstract

A single embedding space that covers text, images, video, and audio lets one index serve every query a user can pose. Embedding models built on vision-language backbones now lead text/image/video retrieval benchmarks but lack audio entirely, while audio-text retrieval is led by specialist systems that serve no other modality. We present the Fusion Embedding family, which adds audio to a frozen vision-language embedding base whose parameters are never updated: generation 1 (fusion-embedding-1) trains only a 16.4M-parameter connector between a frozen audio tower and the frozen base, and generation 2 (fusion-embedding-2) adds modality-gated deep adapters (44.2M parameters) whose branch never executes on text, image, or video inputs: their outputs are bit-for-bit those of the released base, verified after every training run. Because the base already binds text, images, and video, aligning audio to text alone makes audio-image retrieval emerge, with zero paired audio-visual training data. Alongside the recipe we map its design space with controlled negative results (rewriting training captions with an LLM, substituting a leaderboard-stronger audio tower, and widening the connector each reduce retrieval) and with training-protocol findings that we expect to transfer to any frozen decoder-LM embedding backbone. Both generations train in hours on a single GPU. Weights, code, and the evaluation harness are openly released.

Figures

Figures reproduced from arXiv: 2607.18666 by Abdul Basit Tonmoy, Arman Luthra, Kazi Fardinul Hoque, Md. Shahrier Islam Arham.

Figure 1
Figure 1. Figure 1: The Fusion Embedding family. Both generations share the frozen stack: a byte-frozen Qwen3-VL [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Protocol sensitivity (AudioCaps A→T R@10). The same generation-1 checkpoint reads 25.6 points apart under the bare vs. native input format (left), while the bare-trained control (right) shows the native for￾mat inflates nothing. The deployment lesson: evaluate a frozen backbone in its training protocol, or silently forfeit double-digit recall. Both formats are printed ver￾batim in Appendix G. VL-Embedding … view at source ↗
Figure 3
Figure 3. Figure 3: AudioCaps A→T R@10 (883 clips, five￾reference min-rank) versus training-corpus size across the milestones of [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Positioning among unified embedding models on VGGSound-696 cross-modal retrieval (R@10 averaged [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative audio→image retrieval on VGGSound-696 (v0.2 checkpoint; aggregate R@10 0.418, chance 0.014, zero audio–image training pairs). Each row: the query clip’s own frame (left, blue), then the top-5 retrieved frames; green marks the clip’s exact frame, with its rank noted above each row. Retrievals are organized by sound category even when the exact frame ranks lower. Example frames from the VGGSound … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 31 linked inside Pith

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2204.14198

    Pith/arXiv arXiv 2022

  2. [2]

    Jisheng Bai et al. 2024. Audiosetcaps: An enriched audio-caption dataset using automated generation pipeline with large audio and language models. In NeurIPS 2024 Workshop on Audio Imagination. ArXiv:2411.18953

    Pith/arXiv arXiv 2024

  3. [3]

    Simion-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. 2022. Cross modal retrieval with querybank normalisation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). ArXiv:2112.12777

    Pith/arXiv arXiv 2022

  4. [4]

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020. Vggsound: A large-scale audio-visual dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ArXiv:2004.14368

    Pith/arXiv arXiv 2020

  5. [5]

    Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. 2020. Debiased contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2007.00224

    Pith/arXiv arXiv 2020

  6. [6]

    Heinrich Dinkel, Zhiyong Yan, Tianzi Wang, Yongqing Wang, Xingwei Sun, Yadong Niu, Jizhong Liu, Gang Li, Junbo Zhang, and Jian Luan. 2026. Glap: General contrastive audio-text pretraining across domains and languages. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ArXiv:2506.11350

    arXiv 2026

  7. [7]

    Heinrich Dinkel, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun Wang, and Bin Wang. 2024. Scaling up masked audio encoder learning for general audio classification. In Proceedings of Interspeech. ArXiv:2406.06992

    Pith/arXiv arXiv 2024

  8. [8]

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020. Clotho: an audio captioning dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ArXiv:1910.09387. Evaluation split v2.1 from Zenodo record 4783391

    Pith/arXiv arXiv 2020

  9. [9]

    Adnan El Assadi, Isaac Chung, Chenghao Xiao, Roman Solomatin, Animesh Jha, Rahul Chand, Silky Singh, Kaitlyn Wang, Ali Sartaz Khan, Marc Moussa Nasser, Sufen Fong, Pengfei He, Alan Xiao, Ayush Sunil Munot, Aditya Shrivastava, Artem Gazizov, Niklas Muennighoff, and Kenneth Enevoldsen. 2026. MAEB : Massive audio embedding benchmark. arXiv preprint arXiv:2602.16008

    arXiv 2026

  10. [10]

    Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. 2022. Fsd50k: An open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing. ArXiv:2010.00475

    Pith/arXiv arXiv 2022

  11. [11]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind: One embedding space to bind them all. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). ArXiv:2305.05665

    Pith/arXiv arXiv 2023

  12. [12]

    Yuan Gong, Sameer Khurana, Leonid Karlinsky, and James Glass. 2023. Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers. In Proceedings of Interspeech. ArXiv:2307.03183

    Pith/arXiv arXiv 2023

  13. [13]

    Google DeepMind . 2026. Gemini embedding 2: Our first natively multimodal embedding model. https://deepmind.google/models/gemini/embedding/. Commercial API, model id gemini-embedding-2 ; evaluated 2026-07-09

    2026

  14. [16]

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019. Audiocaps: Generating captions for audios in the wild. In Proceedings of NAACL-HLT

    2019

  15. [17]

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2025. Audiocaps 2.0. https://github.com/cdjkim/audiocaps/blob/master/dataset2.0/README.md. Dataset release (February 2025) in the AudioCaps repository: 91 , 256 train / 2 , 475 val / 4 , 875 test pairs

    2025

  16. [18]

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka representation learning. In Advances in Neural Information Processing Systems (NeurIPS). ArXiv:2205.13147

    Pith/arXiv arXiv 2022

  17. [20]

    Jizhong Liu, Gang Li, Junbo Zhang, Heinrich Dinkel, Yongqing Wang, Zhiyong Yan, Yujun Wang, and Bin Wang. 2024. Enhancing automated audio captioning via large language models with optimized audio encoding. In Proceedings of Interspeech. ArXiv:2406.13275

    Pith/arXiv arXiv 2024

  18. [21]

    Plumbley, Yuexian Zou, and Wenwu Wang

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. 2024. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing. ArXiv:2303.17395

    Pith/arXiv arXiv 2024

  19. [22]

    Rui Meng et al. 2025. VLM2Vec-V2 : Advancing multimodal embedding for videos, images, and visual documents. arXiv preprint arXiv:2507.04590. The MMEB-V2 benchmark

    Pith/arXiv arXiv 2025

  20. [24]

    Karol J. Piczak. 2015. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM International Conference on Multimedia

    2015

  21. [25]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML). ArXiv:2103.00020

    Pith/arXiv arXiv 2021

  22. [26]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning (ICML). ArXiv:2212.04356

    Pith/arXiv arXiv 2023

  23. [27]

    Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. Contrastive learning with hard negative samples. In International Conference on Learning Representations (ICLR). ArXiv:2010.04592

    Pith/arXiv arXiv 2021

  24. [28]

    Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision (ECCV) Workshops. ArXiv:1607.01719

    Pith/arXiv arXiv 2016

  25. [30]

    Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. 2023. ONE-PEACE : Exploring one general representation model toward unlimited modalities. arXiv preprint arXiv:2305.11172

    Pith/arXiv arXiv 2023

  26. [31]

    Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, and Zhou Zhao. 2024. Freebind: Free lunch in unified multimodal space via knowledge fusion. In International Conference on Machine Learning (ICML). ArXiv:2405.04883

    Pith/arXiv arXiv 2024

  27. [32]

    Zehan Wang, Ziang Zhang, Minjie Hong, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Shengpeng Ji, Tao Jin, Hengshuang Zhao, and Zhou Zhao. 2025. Omnibind: Large-scale omni multimodal representation via binding spaces. In International Conference on Learning Representations (ICLR). ArXiv:2407.11895

    Pith/arXiv arXiv 2025

  28. [33]

    Chengyue Wu et al. 2024. LLaMA pro: Progressive LLaMA with block expansion. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). ArXiv:2401.02415

    Pith/arXiv arXiv 2024

  29. [34]

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ArXiv:2211.06687

    Pith/arXiv arXiv 2023

  30. [37]

    HaeJun Yoo, Yongseop Shin, Insung Lee, Myoung-Wan Koo, and Du-Seong Chang. 2026. Omni-embed-audio: Leveraging multimodal llms for robust audio-text retrieval. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). ArXiv:2604.18360

    Pith/arXiv arXiv 2026

  31. [38]

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan. 2024. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. In International Conference on Learning Representations (ICLR). ArXiv:2310.01852

    Pith/arXiv arXiv 2024

  32. [39]

    Ge Zhu and Zhiyao Duan. 2024. Cacophony: An improved contrastive audio-text model. IEEE/ACM Transactions on Audio, Speech, and Language Processing. ArXiv:2402.06986

    Pith/arXiv arXiv 2024

  33. [40]

    Proceedings of the 38th International Conference on Machine Learning (ICML) , year =

    Learning Transferable Visual Models From Natural Language Supervision , author =. Proceedings of the 38th International Conference on Machine Learning (ICML) , year =

  34. [41]

    IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year =

    Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation , author =. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year =

  35. [42]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , year =

    WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research , author =. IEEE/ACM Transactions on Audio, Speech, and Language Processing , year =

  36. [43]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , year =

    Cacophony: An Improved Contrastive Audio-Text Model , author =. IEEE/ACM Transactions on Audio, Speech, and Language Processing , year =

  37. [44]

    arXiv preprint arXiv:2503.22104 , year =

    M2D-CLAP: Exploring General-Purpose Audio-Language Representations Beyond CLAP , author =. arXiv preprint arXiv:2503.22104 , year =

    arXiv

  38. [45]

    IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year =

    GLAP: General Contrastive Audio-Text Pretraining Across Domains and Languages , author =. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year =

  39. [46]

    Proceedings of Interspeech , year =

    Scaling up Masked Audio Encoder Learning for General Audio Classification , author =. Proceedings of Interspeech , year =

  40. [47]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    Cross Modal Retrieval with Querybank Normalisation , author =. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  41. [48]

    El Assadi, Adnan and Chung, Isaac and Xiao, Chenghao and Solomatin, Roman and Jha, Animesh and Chand, Rahul and Singh, Silky and Wang, Kaitlyn and Khan, Ali Sartaz and Nasser, Marc Moussa and Fong, Sufen and He, Pengfei and Xiao, Alan and Munot, Ayush Sunil and Shrivastava, Aditya and Gazizov, Artem and Muennighoff, Niklas and Enevoldsen, Kenneth , journal =

  42. [49]

    NeurIPS 2024 Workshop on Audio Imagination , year =

    AudioSetCaps: An Enriched Audio-Caption Dataset Using Automated Generation Pipeline with Large Audio and Language Models , author =. NeurIPS 2024 Workshop on Audio Imagination , year =

    2024

  43. [50]

    Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , year =

    Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval , author =. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , year =

  44. [51]

    arXiv preprint arXiv:2602.18010 , year =

    Scaling Audio-Text Retrieval with Multimodal Large Language Models , author =. arXiv preprint arXiv:2602.18010 , year =

    arXiv

  45. [52]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    ImageBind: One Embedding Space To Bind Them All , author =. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  46. [53]

    2026 , howpublished =

    Gemini Embedding 2: Our First Natively Multimodal Embedding Model , author =. 2026 , howpublished =

    2026

  47. [54]

    International Conference on Learning Representations (ICLR) , year =

    LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment , author =. International Conference on Learning Representations (ICLR) , year =

  48. [55]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Matryoshka Representation Learning , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  49. [56]

    arXiv preprint arXiv:2410.05160 , year =

    VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks , author =. arXiv preprint arXiv:2410.05160 , year =

    Pith/arXiv arXiv

  50. [57]

    2025 , note =

    Meng, Rui and others , journal =. 2025 , note =

    2025

  51. [58]

    arXiv preprint arXiv:2601.04720 , year =

    Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking , author =. arXiv preprint arXiv:2601.04720 , year =

    Pith/arXiv arXiv

  52. [59]

    arXiv preprint arXiv:2503.20215 , year =

    Qwen2.5-Omni Technical Report , author =. arXiv preprint arXiv:2503.20215 , year =

    Pith/arXiv arXiv

  53. [60]

    Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

    Robust Speech Recognition via Large-Scale Weak Supervision , author =. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

  54. [61]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Flamingo: a Visual Language Model for Few-Shot Learning , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  55. [62]

    2024 , note =

    Wu, Chengyue and others , booktitle =. 2024 , note =

    2024

  56. [63]

    arXiv preprint arXiv:2605.08384 , year =

    jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers , author =. arXiv preprint arXiv:2605.08384 , year =

    Pith/arXiv arXiv

  57. [64]

    arXiv preprint arXiv:1807.03748 , year =

    Representation Learning with Contrastive Predictive Coding , author =. arXiv preprint arXiv:1807.03748 , year =

    Pith/arXiv arXiv

  58. [65]

    European Conference on Computer Vision (ECCV) Workshops , year =

    Deep CORAL: Correlation Alignment for Deep Domain Adaptation , author =. European Conference on Computer Vision (ECCV) Workshops , year =

  59. [66]

    Proceedings of NAACL-HLT , year =

    AudioCaps: Generating Captions for Audios in The Wild , author =. Proceedings of NAACL-HLT , year =

  60. [67]

    2025 , note =

    AudioCaps 2.0 , author =. 2025 , note =

    2025

  61. [68]

    IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year =

    Clotho: an Audio Captioning Dataset , author =. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year =

  62. [69]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , year =

    FSD50K: An Open Dataset of Human-Labeled Sound Events , author =. IEEE/ACM Transactions on Audio, Speech, and Language Processing , year =

  63. [70]

    IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year =

    VGGSound: A Large-scale Audio-Visual Dataset , author =. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year =

  64. [71]

    Proceedings of the 23rd ACM International Conference on Multimedia , year =

    ESC: Dataset for Environmental Sound Classification , author =. Proceedings of the 23rd ACM International Conference on Multimedia , year =

  65. [72]

    Proceedings of Interspeech , year =

    Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers , author =. Proceedings of Interspeech , year =

  66. [73]

    Proceedings of Interspeech , year =

    Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding , author =. Proceedings of Interspeech , year =

  67. [74]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Debiased Contrastive Learning , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  68. [75]

    International Conference on Learning Representations (ICLR) , year =

    Contrastive Learning with Hard Negative Samples , author =. International Conference on Learning Representations (ICLR) , year =

  69. [76]

    Wang, Peng and Wang, Shijie and Lin, Junyang and Bai, Shuai and Zhou, Xiaohuan and Zhou, Jingren and Wang, Xinggang and Zhou, Chang , journal =

  70. [77]

    International Conference on Learning Representations (ICLR) , year =

    OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces , author =. International Conference on Learning Representations (ICLR) , year =

  71. [78]

    International Conference on Machine Learning (ICML) , year =

    FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion , author =. International Conference on Machine Learning (ICML) , year =