pith. sign in

arxiv: 2605.29628 · v1 · pith:Y3FB54FRnew · submitted 2026-05-28 · 💻 cs.SD · cs.AI· cs.CL· cs.LG· eess.AS

COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings

Yonggang Zhu , Liting Gao , Aidong Men , Wenwu Wang This is my paper

Pith reviewed 2026-06-29 05:55 UTC · model grok-4.3

classification 💻 cs.SD cs.AIcs.CLcs.LGeess.AS
keywords modality gapCLAPaudio-text embeddingsPLS-SVDconcept decompositionspectral truncationzero-shot learningcontrastive embeddings
0
0 comments X

The pith

A small set of shared concept axes dominates similarity in CLAP embeddings while the mean shift accounts for only part of the modality gap.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper applies partial least squares singular value decomposition to CLAP audio-text embeddings to separate shared concept axes from modality-specific directions. It shows that cosine similarity depends mainly on a few of these shared axes rather than the full embedding space. The conventional mean-vector correction addresses only a fraction of the observed gap. Building on the decomposition, a spectral truncation operation removes the non-shared components and reduces the gap without any model retraining. The resulting embeddings support stronger zero-shot audio captioning via condition swapping and permit large reductions in dimensionality while retaining retrieval performance.

Core claim

COMET reveals that only a small, interpretable subset of axes which captures shared concepts contributes substantially to similarity computation, and that the mean component represents only partially the modality gap. A simple spectral truncation method mitigates the modality gap in a training-free manner, enabling zero-shot audio captioning with condition swapping to approach fully supervised performance without large auxiliary memory banks or expensive computation.

What carries the argument

PLS-SVD transformation that decomposes embeddings into shared-concept axes and modality-specific residual directions.

If this is right

  • Spectral truncation reduces the modality gap without retraining or auxiliary data structures.
  • Zero-shot audio captioning via condition swapping reaches performance levels comparable to fully supervised models.
  • Embeddings can be projected onto far fewer dimensions while preserving retrieval and captioning accuracy.
  • The truncation operates at inference time with negligible extra cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same axis-decomposition technique could be tested on other contrastive multimodal models to locate their modality gaps.
  • Retaining only the shared axes might yield more interpretable embeddings for downstream audio tasks.
  • Dimensionality reduction to the shared subspace could lower storage and inference costs in deployed CLAP systems.

Load-bearing premise

The PLS-SVD transformation isolates axes that are causally responsible for both the similarity scores and the modality gap.

What would settle it

If spectral truncation on the identified axes produces no gain in zero-shot captioning accuracy or retrieval metrics, the claim that these axes control the gap would be falsified.

Figures

Figures reproduced from arXiv: 2605.29628 by Aidong Men, Liting Gao, Wenwu Wang, Yonggang Zhu.

Figure 1
Figure 1. Figure 1: The singular value Σii and its relationship with UV alignment ui ·vi. (a) Σii drops drastically to near-zero values within the top 100 indices, revealing a small, shared core semantic head. (b) ui · vi briefly increases, then drops massively within the top 100 indices. allows each concept to vary freely without interfering with each other. The audio directions vi also have these features. Σ ∈ R C×C is a di… view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of the absolute of the UV matrix |UT V |. The origin is at the upper-left corner. The diagonal elements in the upper-left corner have large activations, while all other parts only have very small activations. Eq. (3) helps to explain the inner product similarity computation. Starting with a positive pair ti · ai , we have: ti · ai ≈ ( XC j=1 tˆijuj )(XC j=1 aˆijvj ) = XC j=1 tˆijaˆij (uj · vj… view at source ↗
Figure 4
Figure 4. Figure 4: Retrieval performance vs number of singular values used on Clotho. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of Xˆ T Xˆ. Note the upper-left diagonal area. where τ is a small temperature parameter. Here, softmax is employed to normalize the similarity weights, which is a nonlinear operation. To build intuition, we first analyze a simplified linear version of PD. Then, we discuss the behavior of the actual PD and provide experimental verifications. A. Linear PD To simplify the original PD into a line… view at source ↗
Figure 6
Figure 6. Figure 6: PLS covariance decomposition and UV alignment for other CLAP. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
read the original abstract

Contrastive Language-Audio Pretraining (CLAP) models are widely used for audio understanding and support modality-agnostic condition swapping in many zero-shot applications. However, their performance is heavily affected by the modality gap between audio and text embeddings. Existing explanations mainly attribute this gap to the cone effect, treating it as a shift between mean embeddings, yet correcting the mean alone yields only limited improvements. Alternative hypotheses, such as information imbalance and dimensionality collapse, have also been proposed, but they remain insufficiently verified and have not been thoroughly studied in the audio domain. Meanwhile, several works attempt to decompose multimodal contrastive embeddings into interpretable concepts, but none explicitly analyze the modality gap from the perspective of concept decomposition. In this work, we introduce COMET (Concept space Organization and Modality gap Explanation with PLS-SVD Transformation), a novel partial least squares singular value decomposition (PLS-SVD) framework for CLAP that unveils a broader perspective of the modality gap. Our framework reveals that only a small, interpretable subset of axes, which captures shared concepts, contributes substantially to similarity computation, and that the mean component represents only partially the modality gap. Building on this insight, we propose a simple spectral truncation method that mitigates the modality gap in a training-free manner. The method enables zero-shot audio captioning with condition swapping to approach fully supervised performance, without requiring large auxiliary memory banks or expensive computation. At the same time, it achieves substantial embedding dimensionality reduction while preserving strong performance on retrieval and audio captioning tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces COMET, a PLS-SVD framework for dissecting concept space in CLAP audio-text embeddings. It claims that only a small, interpretable subset of axes capturing shared concepts substantially contributes to similarity computation, that the mean component only partially represents the modality gap, and that a simple training-free spectral truncation method mitigates the gap. This enables zero-shot audio captioning via condition swapping to approach supervised performance while achieving dimensionality reduction on retrieval and captioning tasks.

Significance. If the decomposition and intervention hold, the work provides an interpretable view of modality gap origins in contrastive audio-text spaces and a practical, training-free mitigation strategy. The emphasis on concept axes rather than mean shift alone, combined with reported gains on captioning without auxiliary memory banks, would be a useful contribution to multimodal embedding analysis in the audio domain.

major comments (3)
  1. [§4, §5] §4 (PLS-SVD analysis) and §5 (spectral truncation): the central claim that the small subset of shared-concept axes 'contributes substantially to similarity computation' rests on covariance maximization but lacks an explicit per-axis decomposition of the cosine similarity (dot product after L2 normalization). Without showing that the retained axes account for the majority of the dot-product mass (e.g., via cumulative contribution plots or controlled ablation of the orthogonal complement), the truncation result remains correlational rather than demonstrating causal dominance.
  2. [§5.2] §5.2 (modality-gap mitigation experiments): the assertion that mean shift 'represents only partially the modality gap' and that truncation removes the remainder requires a controlled comparison where the gap metric (e.g., mean embedding distance or retrieval asymmetry) is measured before/after mean correction alone versus after PLS-SVD truncation. The current evidence does not isolate whether truncation removes gap-related variance or incidental directions.
  3. [Table 2, Figure 4] Table 2 / Figure 4 (zero-shot captioning results): the claim that truncation enables performance 'approaching fully supervised' needs an ablation confirming that the improvement is due to gap reduction rather than general dimensionality reduction or removal of noise; a random-axis truncation baseline of matched rank would strengthen the interpretation.
minor comments (2)
  1. [Eq. 3] Notation for the PLS-SVD transformation (Eq. 3) should explicitly state whether the resulting axes are orthonormal after the SVD step, as this affects the validity of simple truncation.
  2. [Abstract, §5] The abstract states 'substantial embedding dimensionality reduction while preserving strong performance'; the exact retained rank (e.g., top-k value) and the corresponding performance drop should be reported consistently in the main text and tables.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed comments, which help strengthen the empirical grounding of our claims. We address each major comment below and will incorporate the suggested analyses into the revised manuscript.

read point-by-point responses

  1. Referee: [§4, §5] §4 (PLS-SVD analysis) and §5 (spectral truncation): the central claim that the small subset of shared-concept axes 'contributes substantially to similarity computation' rests on covariance maximization but lacks an explicit per-axis decomposition of the cosine similarity (dot product after L2 normalization). Without showing that the retained axes account for the majority of the dot-product mass (e.g., via cumulative contribution plots or controlled ablation of the orthogonal complement), the truncation result remains correlational rather than demonstrating causal dominance.

    Authors: We agree that an explicit per-axis decomposition of the cosine similarity would provide stronger causal evidence. Although PLS-SVD maximizes cross-modal covariance (which directly informs the directions used in similarity), the current manuscript does not include a direct breakdown of dot-product mass. In the revision we will add cumulative contribution plots of the normalized dot product and an ablation that removes the orthogonal complement, quantifying the fraction of similarity mass retained by the shared-concept axes. revision: yes

  2. Referee: [§5.2] §5.2 (modality-gap mitigation experiments): the assertion that mean shift 'represents only partially the modality gap' and that truncation removes the remainder requires a controlled comparison where the gap metric (e.g., mean embedding distance or retrieval asymmetry) is measured before/after mean correction alone versus after PLS-SVD truncation. The current evidence does not isolate whether truncation removes gap-related variance or incidental directions.

    Authors: The referee correctly notes that isolating the incremental effect of truncation beyond mean correction is necessary. The manuscript currently compares truncation to the uncorrected baseline but does not report the intermediate mean-corrected condition. We will add this controlled comparison, measuring mean embedding distance and retrieval asymmetry after mean correction alone and after mean correction followed by PLS-SVD truncation, to demonstrate that truncation removes additional gap-related variance. revision: yes

  3. Referee: [Table 2, Figure 4] Table 2 / Figure 4 (zero-shot captioning results): the claim that truncation enables performance 'approaching fully supervised' needs an ablation confirming that the improvement is due to gap reduction rather than general dimensionality reduction or removal of noise; a random-axis truncation baseline of matched rank would strengthen the interpretation.

    Authors: We acknowledge that a matched-rank random truncation baseline is required to rule out generic dimensionality-reduction effects. The current results compare spectral truncation only to the full embedding and to mean correction. In the revision we will include a random-axis truncation baseline at the same retained rank in both Table 2 and Figure 4, allowing direct comparison of performance gains attributable to concept-axis selection versus random selection. revision: yes

Circularity Check

0 steps flagged

No circularity; derivation is self-contained empirical analysis

full rationale

The paper applies standard PLS-SVD to decompose CLAP embeddings into axes, identifies a subset as shared concepts via covariance maximization, and proposes post-hoc spectral truncation as a training-free mitigation. No equations or steps reduce a claimed prediction to a fitted parameter by construction, nor does any load-bearing claim rest on self-citation of an unverified uniqueness result. The framework is externally falsifiable via retrieval and captioning metrics, and the central interpretation (small axes dominate similarity) is presented as an empirical observation rather than a definitional identity. This matches the default case of an independent analysis with no circular reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides insufficient detail to identify any free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5832 in / 1145 out tokens · 26663 ms · 2026-06-29T05:55:20.141269+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 8 canonical work pages · 2 internal anchors

  1. [1]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwalet al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  2. [2]

    Natural language supervision for general-purpose audio representations,

    B. Elizalde, S. Deshmukh, and H. Wang, “Natural language supervision for general-purpose audio representations,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2024, pp. 336–340

    2024

  3. [3]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2023, pp. 1–5. 4ATR HTSAT+RoBERTa ASC PT mask25 T2AR1 32.8 A2TR1 44.5.pt 12 ...

    2023

  4. [4]

    Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio- language multimodal research,

    X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhaoet al., “Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio- language multimodal research,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  5. [5]

    Auto-acd: A large-scale dataset for audio-language representation learning,

    L. Sun, X. Xu, M. Wu, and W. Xie, “Auto-acd: A large-scale dataset for audio-language representation learning,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 5025–5034

    2024

  6. [6]

    Audiosetcaps: An enriched audio-caption dataset using automated gener- ation pipeline with large audio and language models,

    J. Bai, H. Liu, M. Wang, D. Shi, W. Wang, M. D. Plumbleyet al., “Audiosetcaps: An enriched audio-caption dataset using automated gener- ation pipeline with large audio and language models,”IEEE Transactions on Audio, Speech and Language Processing, 2025

    2025

  7. [7]

    A knowledge distillation approach to im- proving language-based audio retrieval models,

    P. Primus and G. Widmer, “A knowledge distillation approach to im- proving language-based audio retrieval models,” DCASE2024 Challenge, Tech. Rep., June 2024

    2024

  8. [8]

    Aistat lab system for dcase 2025 task6: Language-based audio retreival,

    H. J. Kim, H. Y . Choi, K. Choi, E. Choi, and C. Lim, “Aistat lab system for dcase 2025 task6: Language-based audio retreival,” DCASE2025 Challenge, Tech. Rep., June 2025

    2025

  9. [9]

    Recap: Retrieval-augmented audio captioning,

    S. Ghosh, S. Kumar, C. K. R. Evuru, R. Duraiswami, and D. Manocha, “Recap: Retrieval-augmented audio captioning,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing. IEEE, 2024, pp. 1161–1165

    2024

  10. [10]

    Diffusion-based diverse audio captioning with retrieval-guided langevin dynamics,

    Y . Zhu, A. Men, and L. Xiao, “Diffusion-based diverse audio captioning with retrieval-guided langevin dynamics,”Information Fusion, vol. 114, p. 102643, 2025

    2025

  11. [11]

    Zero-shot diverse audio captioning with diffusion models,

    Y . Zhu, Y . Zhang, L. Xiao, W. Wang, and A. Men, “Zero-shot diverse audio captioning with diffusion models,”Knowledge-Based Systems, vol. 335, p. 115205, 2026

    2026

  12. [12]

    Audioldm: text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandicet al., “Audioldm: text-to-audio generation with latent diffusion models,” inProceedings of the 40th International Conference on Machine Learning, 2023, pp. 21 450–21 474

    2023

  13. [13]

    AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,

    H. Liu, Y . Yuan, X. Liu, X. Mei, Q. Kong, Q. Tianet al., “AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  14. [14]

    Weakly-supervised automated audio captioning via text only training,

    T. Kouzelis and V . Katsouros, “Weakly-supervised automated audio captioning via text only training,” inDetection and Classification of Acoustic Scenes and Events (DCASE) Workshop, 2023, pp. 81–85

    2023

  15. [15]

    Training audio captioning models without audio,

    S. Deshmukh, B. Elizalde, D. Emmanouilidou, B. Raj, R. Singh, and H. Wang, “Training audio captioning models without audio,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2024, pp. 371–375

    2024

  16. [16]

    Zero-shot audio captioning using soft and hard prompts,

    Y . Zhang, X. Xu, R. Du, H. Liu, Y . Dong, Z.-H. Tanet al., “Zero-shot audio captioning using soft and hard prompts,”IEEE Transactions on Audio, Speech and Language Processing, 2025

    2025

  17. [17]

    Drcap: Decoding clap latents with retrieval-augmented generation for zero-shot audio captioning,

    X. Li, W. Chen, Z. Ma, X. Xu, Y . Liang, Z. Zhenget al., “Drcap: Decoding clap latents with retrieval-augmented generation for zero-shot audio captioning,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2025, pp. 1–5

    2025

  18. [18]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,

    V . W. Liang, Y . Zhang, Y . Kwon, S. Yeung, and J. Y . Zou, “Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 17 612–17 625, 2022

    2022

  19. [19]

    Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models,

    S. Schrodi, D. T. Hoffmann, M. Argus, V . Fischer, and T. Brox, “Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models,” inInternational Con- ference on Learning Representations, 2025

    2025

  20. [20]

    Decipher the modality gap in multi- modal contrastive learning: From convergent representations to pairwise alignment,

    L. Yi, R. Douady, and C. Chen, “Decipher the modality gap in multi- modal contrastive learning: From convergent representations to pairwise alignment,”arXiv preprint arXiv:2510.03268, 2025

    work page arXiv 2025

  21. [21]

    Decap: Decoding CLIP latents for zero-shot captioning via text-only training,

    W. Li, L. Zhu, L. Wen, and Y . Yang, “Decap: Decoding CLIP latents for zero-shot captioning via text-only training,” inInternational Conference on Learning Representations, 2023

    2023

  22. [22]

    Diffgap: A lightweight diffusion module in contrastive space for bridging cross-model gap,

    S. Mo, Z. Chen, F. Bao, and J. Zhu, “Diffgap: A lightweight diffusion module in contrastive space for bridging cross-model gap,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2025, pp. 1–5

    2025

  23. [23]

    Diffusion-link: Diffusion probabilistic model for bridging the audio-text modality gap,

    K. Nam, J. Choi, H. Lee, J. Heo, and J. S. Chung, “Diffusion-link: Diffusion probabilistic model for bridging the audio-text modality gap,” arXiv preprint arXiv:2510.11330, 2025

    work page arXiv 2025

  24. [24]

    Pull it together: Reducing the modality gap in contrastive learning,

    A. Sofer, Y . Goldman, and S. E. Chazan, “Pull it together: Reducing the modality gap in contrastive learning,” inProc. Interspeech 2025, 2025, pp. 196–200

    2025

  25. [25]

    Interpreting clip with sparse linear concept embeddings (splice),

    U. Bhalla, A. Oesterling, S. Srinivas, F. P. Calmon, and H. Lakkaraju, “Interpreting clip with sparse linear concept embeddings (splice),” in Advances in Neural Information Processing Systems, A. Globerson, 13 L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczaket al., Eds., vol. 37. Curran Associates, Inc., 2024, pp. 84 298–84 328

    2024

  26. [26]

    Quantifying structure in clip embeddings: A statistical framework for concept interpretation,

    J. Zhao, C. Li, F. Sala, and K. Rohe, “Quantifying structure in clip embeddings: A statistical framework for concept interpretation,”arXiv preprint arXiv:2506.13831, 2025

    work page arXiv 2025

  27. [27]

    Scocca: Multi-modal sparse concept decomposition via canonical correlation analysis,

    E. Gordon, M. Y . Levi, and G. Gilboa, “Scocca: Multi-modal sparse concept decomposition via canonical correlation analysis,”arXiv preprint arXiv:2603.13884, March 2026

    work page arXiv 2026

  28. [28]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9729–9738

    2020

  29. [29]

    Pengi: An audio language model for audio tasks,

    S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,”Advances in Neural Information Pro- cessing Systems, vol. 36, pp. 18 090–18 108, 2023

    2023

  30. [30]

    RFM-Editing: Rectified Flow Matching for Text-guided Audio Editing

    L. Gao, Y . Yuan, Y . Chen, Y . Cheng, Z. Li, J. Wenet al., “Rfm-editing: Rectified flow matching for text-guided audio editing,”arXiv preprint arXiv:2509.14003, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [31]

    A holistic approach to unifying automatic concept extraction and concept importance estimation,

    T. Fel, V . Boutin, L. B´ethune, R. Cad `ene, M. Moayeri, L. And ´eolet al., “A holistic approach to unifying automatic concept extraction and concept importance estimation,”Advances in Neural Information Processing Systems, vol. 36, pp. 54 805–54 818, 2023

    2023

  32. [32]

    Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav),

    B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegaset al., “Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav),” inInternational Conference on Machine Learning. PMLR, 2018, pp. 2668–2677

    2018

  33. [33]

    Craft: Concept recursive activation factorization for explainability,

    T. Fel, A. Picard, L. Bethune, T. Boissin, D. Vigouroux, J. Colinet al., “Craft: Concept recursive activation factorization for explainability,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2711–2721

    2023

  34. [34]

    Multi-dimensional concept discovery (mcd): a unifying framework with completeness guarantees,

    J. Vielhaben, S. Bl ¨ucher, and N. Strodthoff, “Multi-dimensional concept discovery (mcd): a unifying framework with completeness guarantees,” Transactions on machine learning research, no. 5, pp. 1–25, 2023

    2023

  35. [35]

    Concept bottleneck models,

    P. W. Koh, T. Nguyen, Y . S. Tang, S. Mussmann, E. Pierson, B. Kim et al., “Concept bottleneck models,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 5338–5348

    2020

  36. [36]

    Label-free concept bottleneck models,

    T. Oikarinen, S. Das, L. M. Nguyen, and T.-W. Weng, “Label-free concept bottleneck models,” inInternational Conference on Learning Representations, 2023

    2023

  37. [37]

    Post-hoc concept bottleneck models,

    M. Yuksekgonul, M. Wang, and J. Zou, “Post-hoc concept bottleneck models,” inInternational Conference on Learning Representations, 2023

    2023

  38. [38]

    Transformation of audio embeddings into interpretable, concept-based representations,

    A. Zhang, E. Thomaz, and L. Lu, “Transformation of audio embeddings into interpretable, concept-based representations,” in2025 International Joint Conference on Neural Networks. IEEE, 2025, pp. 1–8

    2025

  39. [39]

    Pre-trained vision-language models learn discoverable visual concepts,

    Y . Zang, T. Yun, H. Tan, T. Bui, and C. Sun, “Pre-trained vision-language models learn discoverable visual concepts,”Transactions on Machine Learning Research, 2025

    2025

  40. [40]

    Diagnosing and rectifying vision models using language,

    Y . Zhang, J. Z. HaoChen, S.-C. Huang, K.-C. Wang, J. Zou, and S. Yeung, “Diagnosing and rectifying vision models using language,” inInternational Conference on Learning Representations, 2023

    2023

  41. [41]

    Towards understanding the modality gap in clip,

    P. Shi, M. C. Welle, M. Bj¨orkman, and D. Kragic, “Towards understanding the modality gap in clip,” inICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023

    2023

  42. [42]

    Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data,

    Y . Zhang, E. Sui, and S. Yeung, “Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data,” inInternational Conference on Learning Representations, 2024

    2024

  43. [43]

    Closing the modality gap for mixed modality search,

    B. Li, Y . Zhang, X. Wang, W. Liang, L. Schmidt, and S. Yeung- Levy, “Closing the modality gap for mixed modality search,” inSocially Responsible and Trustworthy Foundation Models at NeurIPS 2025, 2025

    2025

  44. [44]

    Explaining and mitigating the modality gap in contrastive multimodal learning,

    C. Yaras, S. Chen, P. Wang, and Q. Qu, “Explaining and mitigating the modality gap in contrastive multimodal learning,” inConference on Parsimony and Learning. PMLR, 2025, pp. 1365–1387

    2025

  45. [45]

    Closing the modality gap enables novel multimodal learning applications,

    E. Grassucci, G. Cicchetti, and D. Comminiello, “Closing the modality gap enables novel multimodal learning applications,” inSecond Workshop on Representational Alignment at ICLR 2025, 2025

    2025

  46. [46]

    Accept the modality gap: An exploration in the hyperbolic space,

    S. Ramasinghe, V . Shevchenko, G. Avraham, and A. Thalaiyasingam, “Accept the modality gap: An exploration in the hyperbolic space,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 27 263–27 272

    2024

  47. [47]

    It’s not a modality gap: Characteriz- ing and addressing the contrastive gap,

    A. Fahim, A. Murphy, and A. Fyshe, “It’s not a modality gap: Characteriz- ing and addressing the contrastive gap,”arXiv preprint arXiv:2405.18570, 2024

    work page arXiv 2024

  48. [48]

    Understanding contrastive representation learning through alignment and uniformity on the hypersphere,

    T. Wang and P. Isola, “Understanding contrastive representation learning through alignment and uniformity on the hypersphere,” inInternational Conference on Machine Learning. PMLR, 2020, pp. 9929–9939

    2020

  49. [49]

    I0t: Embedding standard- ization method towards zero modality gap,

    N. M. An, E. Kim, J. Thorne, and H. Shim, “I0t: Embedding standard- ization method towards zero modality gap,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025, pp. 27 182–27 199

    2025

  50. [50]

    Mitigate the gap: Improving cross-modal alignment in clip,

    S. Eslami and G. de Melo, “Mitigate the gap: Improving cross-modal alignment in clip,” inInternational Conference on Learning Representa- tions, 2025

    2025

  51. [51]

    Diffusion bridge: leveraging diffusion model to reduce the modality gap between text and vision for zero-shot image captioning,

    J. R. Lee, Y . Shin, G. Son, and D. Hwang, “Diffusion bridge: leveraging diffusion model to reduce the modality gap between text and vision for zero-shot image captioning,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 4050–4059

    2025

  52. [52]

    Text-only training for image captioning using noise-injected clip,

    D. Nukrai, R. Mokady, and A. Globerson, “Text-only training for image captioning using noise-injected clip,” inFindings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 4055–4063

    2022

  53. [53]

    I can’t believe there’s no images! learning visual tasks using only language supervision,

    S. Gu, C. Clark, and A. Kembhavi, “I can’t believe there’s no images! learning visual tasks using only language supervision,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2672–2683

    2023

  54. [54]

    Zero-shot audio cap- tioning with audio-language model guidance and audio context keywords,

    L. Salewski, S. Fauth, A. Koepke, and Z. Akata, “Zero-shot audio cap- tioning with audio-language model guidance and audio context keywords,” in37th Conference on Neural Information Processing Systems (NeurIPS

  55. [55]

    - ML for Audio Workshop, 2023

    2023

  56. [56]

    Zero-shot audio captioning via audibility guidance,

    T. Shaharabany, A. Shaulov, and L. Wolf, “Zero-shot audio captioning via audibility guidance,”arXiv preprint arXiv:2309.03884, 2023

    work page arXiv 2023

  57. [57]

    Clotho: An audio caption- ing dataset,

    K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio caption- ing dataset,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2020, pp. 736–740

    2020

  58. [58]

    Audiocaps: Generating captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generating captions for audios in the wild,” inProceedings of NAACL-HLT, 2019

    2019

  59. [59]

    Sound- vecaps: Improving audio generation with visually enhanced captions,

    Y . Yuan, D. Jia, X. Zhuang, Y . Chen, Z. Chen, Y . Wanget al., “Sound- vecaps: Improving audio generation with visually enhanced captions,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2025, pp. 1–5

    2025

  60. [60]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  61. [61]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wanget al., “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022

    2022

  62. [62]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuanget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  63. [63]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei et al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  64. [64]

    Minimum bayes-risk decoding for statistical machine translation,

    S. Kumar and B. Byrne, “Minimum bayes-risk decoding for statistical machine translation,” inProceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, 2004, pp. 169–176

    2004

  65. [65]

    HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection,

    K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, “HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection,” inICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2022, pp. 646–650

    2022

  66. [66]

    BLEU: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” inProceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

    2002

  67. [67]

    METEOR: An automatic metric for mt evalu- ation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “METEOR: An automatic metric for mt evalu- ation with improved correlation with human judgments,” inProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005, pp. 65–72

    2005

  68. [68]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” inText Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81

    2004

  69. [69]

    CIDEr: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “CIDEr: Consensus- based image description evaluation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2015, pp. 4566– 4575

    2015

  70. [70]

    Spice: Semantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” inEuropean Conference on Computer Vision. Springer, 2016, pp. 382–398

    2016

  71. [71]

    Improved image captioning via policy gradient optimization of spider,

    S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Improved image captioning via policy gradient optimization of spider,” inProceedings of the IEEE International Conference on Computer Vision, 2017, pp. 873– 881

    2017