Pith. sign in

REVIEW 3 major objections 5 minor 76 references

Aligning Multimodal Representations through an Information Bottleneck

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that the modality gap in contrastive multimodal learning is caused by modality-specific information retained in the representations, and that a single added term — the expected squared distance between paired encoders'…

desk verdict Useful empirical study of nuisance information and alignment, but the IB derivation of the proposed regularizer is invalid and the final loss is a known alignment term. read the letter →

arxiv 2506.04870 v1 pith:P34XQSAD submitted 2025-06-05 cs.LG

Antonio Almudévar , José Miguel Hernández-Lobato , Sameer Khurana , Ricard Marxer , Alfonso Ortega This is my paper
classification cs.LG
keywords modalitygapcontrastivelearninginformationbottleneckrepresentationalignmentnuisancemultimodalInfoNCEminimalsufficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contrastively trained multimodal models leave a systematic gap between the representation spaces of, say, images and text, and this paper argues that the cause is modality-specific information — 'nuisances' — that encoders retain alongside the information shared by both modalities. The paper proves that perfectly aligned representations must be free of nuisance information, then derives a two-modality Information Bottleneck objective that, under a Gaussian assumption, collapses into one extra loss term: the expected squared distance between the two encoders' outputs on matched pairs, added to InfoNCE. In controlled experiments on disentanglement datasets, the term lowers a linear-probe estimate of nuisance information and raises Centered Kernel Alignment. In a Q-Former image-captioning system, moderate strengths of the term improve CIDEr and BLEU-4 scores with only a small retrieval cost, indicating a simple, modality-agnostic way to close the modality gap.

What carries the argument

The load-bearing object is the two-modality Information Bottleneck objective, in which each modality's task is the other modality's input: maximize $I(Z_\alpha; X_\beta) - \beta I(Z_\alpha; X_\alpha)$ jointly for $\alpha$ and $\beta$. The tractable mechanism that carries the argument is the chain of bounds from Section 4: minimizing InfoNCE maximizes a lower bound on $I(Z_\alpha; X_\beta)$, giving sufficiency; minimizing the expected KL divergence between the two encoders' posterior distributions on matched pairs upper-bounds $I(Z_\alpha; X_\alpha)$, giving minimality. When both posteriors are taken to be spherical Gaussians with one shared variance, $\sigma^2 I$, the KL term is proportional to the mean-squared difference of the encoder means, which produces the added loss $L_{\mathcal{M}}$; this is the object that turns the abstract IB principle into a one-line, modality-agnostic penalty.

What would settle it

Run the controlled toy experiments (DSprites, MPI3D, Shapes3D) with the proposed loss at fixed temperature, sweeping $\beta$ while measuring both a linear-probe estimate of $I(Z_\alpha; N_\alpha)$ and CKA alignment. The paper's mechanism predicts that nuisance information drops monotonically as $\beta$ rises and that the alignment gain is driven by that drop. Observing a regime in which alignment improves while probe-recoverable nuisance information stays flat or rises would falsify the causal story; so would showing that the term changes alignment or downstream performance on a no-nuisance dataset where both modalities are deterministic functions of the same essence.

Watch

Extended reading notes

Core claim

The paper's central claim is that the modality gap is not an optimization artifact but an information-theoretic consequence of contrastive learning: InfoNCE maximizes the mutual information that representations share (sufficiency) while imposing nothing about what each representation keeps that the other modality does not have (minimality), so nuisances are preserved and the two spaces drift apart. Theorem 2 states the necessity direction: if $Z_\alpha$ and $Z_\beta$ are aligned in the kernel sense of equation (2), then $I(Z_\alpha; N_\alpha) = I(Z_\beta; N_\beta) = 0$. To reach such representations the paper minimizes $I(Z_\alpha; X_\alpha)$ through the variational upper bound $I(Z_\alpha; X_\alpha) \le \mathbb{E}_{p(x_\alpha,x_\beta)}[D_{\mathrm{KL}}(p_{\theta_\alpha}(z|x_\alpha)\,\|\,p_{\theta_\beta}(z|x_\beta))]$, and under the assumption that both posteriors are spherical Gaussians with a shared fixed variance this becomes the squared-distance regularizer $L_{\mathcal{M}} = \mathbb{E}[\|\mu_{\theta_\alpha}(x_\alpha) - \mu_{\theta_\beta}(x_\beta)\|^2]$, giving the total loss $L = L_{\mathrm{InfoNCE}} + \beta L_{\mathcal{M}}$. Empirically the paper shows nuisance information is present across datasets and architectures, is negatively correlated with alignment, is reduced by depth, by higher temperatures, and by the new term, and its removal is what improves image captioning.

Load-bearing premise

The derivation of the proposed loss assumes each encoder's representation distribution is a spherical Gaussian with the same fixed variance — with the deterministic encoder output taken as the Gaussian's mean — so if the true conditional distributions are not equal-variance Gaussians, the regularizer is a heuristic alignment penalty rather than a genuine Information Bottleneck term.

Editorial extensions

If this is right

  • Any contrastively trained multimodal model can lower its modality gap by appending $\beta\,\mathbb{E}[\|\mu_{\theta_\alpha}(x_\alpha) - \mu_{\theta_\beta}(x_\beta)\|^2]$ to InfoNCE, with $\beta$ explicitly trading sufficiency (retrieval) against minimality (alignment).
  • In the Q-Former image-captioning experiments, beta values of 0.01 to 0.1 raise CIDEr from 91.7 to 93.0 and BLEU@4 from 28.6 to 29.4 while costing at most 1.2 points of image-to-text retrieval, whereas beta = 0.3 damages both captioning and retrieval.
  • The regularizer is modality-agnostic and computationally cheap, so it can substitute for task-specific alignment losses such as image-text matching in vision-language pipelines.
  • The paper's 'information homeostasis' finding implies that a trainable InfoNCE temperature will fight the regularizer by dropping to preserve nuisance information; fixing the temperature makes the term effective, and monitoring it is necessary in practice.
  • Deeper encoders and higher fixed temperatures are additional levers that remove nuisances on their own, independent of the proposed loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper demonstrates the benefit only for image captioning, an immediate testable extension is whether the same term improves other cross-modal generation tasks — text-to-image, visual question answering, audio captioning — where a decoder consumes one modality and nuisances leak through the representation space; the mechanism is task-independent though the experiments are not.
  • Appendix B's equivalence between the regularizer and a numerator/denominator temperature mismatch in InfoNCE suggests the penalty could be reimplemented as an adaptive temperature schedule, making it essentially free in existing codebases; the paper notes the equivalence but does not pursue this implementation.
  • The necessity theorem suggests a cheap audit for nuisance retention: on a model whose two modalities are highly aligned by CKA, a linear probe on known nuisance labels should fail; conversely, misalignment becomes a warning sign that modality-specific information survived, which could serve as a diagnostic when nuisances are not known in advance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that the modality gap in contrastive multimodal representation learning is caused by modality-specific nuisance information retained in the representations, and proposes a regularization term L_M = E[||mu_theta_alpha(x_alpha) - mu_theta_beta(x_beta)||^2] added to InfoNCE (Eq. 20) to remove nuisances and increase alignment. It formalizes essence and nuisance variables, characterizes minimal sufficient representations, derives an upper bound on I(Z_alpha; X_alpha) that reduces to L_M under spherical-Gaussian posteriors, and validates the approach in toy experiments on DSprites, MPI3D, and Shapes3D, as well as in image captioning with a Q-Former on COCO.

Significance. The paper addresses a real and timely problem: contrastive multimodal models exhibit a modality gap, and the authors propose a simple, modality-agnostic regularizer that improves alignment and, at small beta, captioning quality. The controlled toy experiments in Section 5 are a clear strength: they measure nuisance information via a linear-probe lower bound, vary encoder depth, temperature, and beta, and show a consistent negative correlation between nuisance information and CKA alignment, as well as a plausible trade-off under beta. The release of code supports reproducibility of the empirical claims. If the theoretical derivation were valid, the IB framing would give a principled justification for an alignment penalty that is otherwise close to existing losses such as the alignment term of Wang and Isola (2020). However, as detailed below, the key upper-bound proof is incorrect, so the main contribution at present is the empirical demonstration plus the information-homeostasis observation rather than a new theoretical foundation.

major comments (3)
  1. [Appendix A.5 / Eq. (17) and Eq. (28)] The proof of the upper bound in Eq. (17) is invalid. Eq. (28) rewrites the cross term as -E_{p(x_alpha,x_beta|z)} D_KL(p_theta_alpha(z) || p_theta_beta(z|x_beta)), but the log-ratio in the preceding integral is evaluated at z drawn from p_theta_alpha(z|x_alpha), whereas the KL term is an integral over the marginal p_theta_alpha(z) and depends on x_beta only through the conditioning argument; these two objects are not equal. The failure is not a technicality: take X_beta = Y and X_alpha = (Y, N) with N independent of Y, and set both encoders to p_theta_alpha(z|x_alpha) = p_theta_beta(z|x_beta) = N(z; Y, sigma^2 I). Then the RHS of Eq. (17) is 0, while I(Z_alpha; X_alpha) >= I(Z_alpha; Y) > 0 for non-degenerate Y. Thus Eq. (17) is false as stated, and minimizing L_M is not shown to minimize I(Z_alpha; X_alpha). The empirical results can stand without this bound, but the paper's central theoretical claim that L_M is an Information Bottleneck regularizer is not established and must either be repaired or explicitly replaced by a heuristic-alignment interpretation.
  2. [Section 4.2 / Eq. (18)] The reduction of the KL divergence to the squared-ell2 penalty L_M assumes the encoder posteriors are isotropic Gaussians with a shared, fixed variance sigma^2 I, but the experiments use deterministic encoders with no noise injection. Therefore the variational-approximation derivation does not apply to the actual models; L_M is, as implemented, an alignment penalty on deterministic embeddings. The authors should either inject Gaussian noise to make the posterior model operational and report sigma, or present L_M as a heuristic regularization whose IB interpretation is only suggestive. Without this, the claimed connection to the Information Bottleneck principle is unsupported at the implementation level.
  3. [Section 3.2 / Theorem 2 and Appendix A.3] Theorem 2 is stated informally, and the proof does not establish the claimed implication. The assertion that I(Z_alpha; N_alpha) != 0 implies the existence of a surjective f with Z_beta = f(Z_alpha) is not derived from any stated assumption, and the argument that distinct nuisance values prevent perfect CKA is only heuristic. The proof itself concedes the finite-sample caveat. Since minimality-to-alignment is the conceptual bridge of the paper, the statement should either be given a precise formulation with explicit assumptions under which it holds, or be labeled as intuition supported by the correlational experiments in Section 5.2.
minor comments (5)
  1. [Section 6.1 / Table 3] The paper should report the number of random seeds and define the error bars; the captioning improvement (91.7 to 93.0 CIDEr) is modest and comes with a retrieval drop, so the word 'substantially' in the text overstates the effect.
  2. [Appendix B] The temperature-mismatch equivalence is an interesting observation; consider summarizing it in the main text because it reinforces that the regularizer acts as an adaptive temperature adjustment rather than an IB penalty.
  3. [Section 7] The sentence 'In the the opposite direction' contains a duplicated 'the'.
  4. [Table 3 / Section 4.3] The notation '0.01LM' conflates the coefficient beta with the loss name; use 'beta = 0.01' with L_M for clarity.
  5. [Figure 5] The x-axis label uses a hat notation for I(Z_alpha; N_alpha) but the text calls it URR; make explicit that this is the lower-bound proxy normalized by H(N_alpha).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: L_M is an explicit alignment penalty and the central empirical claims are tested on controlled toy data and held-out COCO captioning/retrieval.

full rationale

The paper's derivation chain is not circular in the sense of fitting an input and renaming it a prediction, nor does it rely on a load-bearing self-citation. The proposed regularizer L_M (Eq. 18) is an explicit expected squared-distance penalty between the two encoders' mean embeddings; the paper does not tune a parameter to a target result and then present that result as a prediction. The claims that contrastive losses retain nuisance information, that retained nuisance information correlates negatively with CKA alignment, and that adding beta L_M improves image captioning are evaluated on controlled toy datasets and on held-out COCO captioning/retrieval, giving the central claims independent empirical content. The main concerns in the paper are correctness and novelty, not circularity: the proof of Eq. 17 in Appendix A.5 relies on an identity (Eq. 28) that appears false, and the spherical-Gaussian assumption in Section 4.2 and Appendix A.6 reduces the KL upper bound to a mean-square heuristic, so the IB derivation may not establish nuisance removal; but a false or heuristic derivation is not the same as a circular one. The only self-citation (Cuervo and Marxer 2024, Section 1) is a background scaling-law citation with independent empirical content and carries no load in the argument. Overlap with Federici et al. (2020) is a novelty concern, not circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 2 invented entities

The central derivation rests on a generative model in which each input is the pair (essence, nuisance), on a Gaussian conditional-encoder assumption, and on an unstated minimality of the essence used in Lemma 1. The only freely chosen number with real effect is beta; sigma^2 is folded into it. These assumptions are not validated against real data, but they are the price of the tractable loss.

free parameters (2)
  • beta (trade-off weight) = swept over {0, 0.01, 0.03, 0.1, 0.3, 1.0, 3.0}; best captioning at 0.1
    Controls the balance between sufficiency (InfoNCE) and minimality (alignment penalty). The paper sweeps it rather than fitting, but the choice of sweep range and the task-dependent optimum make it a free parameter of the method.
  • sigma^2 (posterior variance in Gaussian representation) = implicitly 1, absorbed into beta
    The Gaussian assumption in Appendix A.6 introduces a variance sigma^2; the final loss drops the 1/(2 sigma^2) constant into beta, so sigma^2 is never estimated. It is a free modeling choice that affects the interpretation of the bound.
assumptions (3)
  • domain assumption Data is generated as X_alpha = (Y, N_alpha), X_beta = (Y, N_beta) with Y the common essence and N_alpha, N_beta independent of Y and of each other
    Definitions 1-2 posit exact Markov-chain structure. Real multimodal data may not decompose this cleanly; common nuisance correlations (e.g., lighting in both image and caption) could violate the independence of N_alpha and X_beta.
  • ad hoc to paper Representation posteriors are spherical Gaussians with shared constant variance sigma^2 I
    Section 4.2/Appendix A.6. This makes the KL upper bound proportional to ||mu_theta_alpha(x_alpha) - mu_theta_beta(x_beta)||^2. Deterministic encoders do not model a posterior, so this is a tractability assumption, not a learned property.
  • ad hoc to paper Essences are minimal sufficient statistics (unique up to bijection)
    Lemma 1's proof (Appendix A.1) assumes Y and Y' are minimal sufficient statistics, but Definition 1 only requires two Markov chains (sufficiency). Without an explicit minimality condition, extra independent noise can be added to Y and still satisfy the chains, breaking one-to-one equivalence.
invented entities (2)
  • Essence Y
    purpose: Latent common information between a positive pair of modalities
    Defined through Markov chains in Definition 1; never observed. The paper says the goal is not to recover Y but the partition it induces.
  • Nuisance N_alpha
    purpose: Modality-specific information in modality alpha that is absent from the other modality
    Defined via entropy equalities in Definition 2; not directly measurable. Estimated only through a lower bound (linear classifier URR) in experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Aligning Multimodal Representations through an Information Bottleneck." pith.science (2026). https://pith.science/paper/P34XQSAD

@misc{pith2026250604870,
  author       = {Pith},
  title        = {Pith review of: Aligning Multimodal Representations through an Information Bottleneck},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P34XQSAD}},
  note         = {Machine review of arXiv:2506.04870}
}
read the original abstract

Contrastive losses have been extensively used as a tool for multimodal representation learning. However, it has been empirically observed that their use is not effective to learn an aligned representation space. In this paper, we argue that this phenomenon is caused by the presence of modality-specific information in the representation space. Although some of the most widely used contrastive losses maximize the mutual information between representations of both modalities, they are not designed to remove the modality-specific information. We give a theoretical description of this problem through the lens of the Information Bottleneck Principle. We also empirically analyze how different hyperparameters affect the emergence of this phenomenon in a controlled experimental setup. Finally, we propose a regularization term in the loss function that is derived by means of a variational approximation and aims to increase the representational alignment. We analyze in a set of controlled experiments and real-world applications the advantages of including this regularization term.

Figures

Figures reproduced from arXiv: 2506.04870 by the authors.

Figure 1
Figure 1. Different modalities usually contain different in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Diagram of the inputs, essence and nuisances [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. URR (y-axis) for different values of temperature [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: URR (y-axis) for different number of layers of the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 6
Figure 6. Figure 6: Relative change (with respect to the case in which [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Relative change (with respect to the case in which [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 9
Figure 9. Figure 9: shows image retrievals obtained from combining image and text representations from the Q-Former trained with β = 0.01. Examples including other loss functions are found in Appendix E, showing that those not encouraging alignment, result in worse multimodal retrievals. …
Figure 8
Figure 8. Figure 8: Captions generated by some of the trained models. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: Captions generated by some of the trained models. Numbers correspondence is the same as in Table [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

76 extracted references · 37 canonical work pages

  1. [1]

    and Soatto, S

    Achille, A. and Soatto, S. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19 0 (50): 0 1--34, 2018 a

    work page 2018

  2. [2]

    and Soatto, S

    Achille, A. and Soatto, S. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 0 (12): 0 2897--2905, 2018 b

    work page 2018

  3. [3]

    A., Fischer, I., Dillon, J

    Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016

    arXiv 2016

  4. [4]

    D., and Buchwalter, W

    Bachman, P., Hjelm, R. D., and Buchwalter, W. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019

    work page 2019

  5. [5]

    Revisiting model stitching to compare neural representations

    Bansal, Y., Nakkiran, P., and Barak, B. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34: 0 225--236, 2021

    2021

  6. [6]

    Representation learning: A review and new perspectives

    Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35 0 (8): 0 1798--1828, 2013

    2013

  7. [7]

    and Kim, H

    Burgess, C. and Kim, H. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018

    work page 2018

  8. [8]

    F., Hong, H., Yamins, D

    Cadieu, C. F., Hong, H., Yamins, D. L., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., and DiCarlo, J. J. Deep neural networks rival the representation of primate it cortex for core visual object recognition. PLoS computational biology, 10 0 (12): 0 e1003963, 2014

    work page 2014

Show all 76 references
  1. [9]

    A simple framework for contrastive learning of visual representations

    Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.\ 1597--1607. PMLR, 2020 a

    2020

  2. [10]

    Uniter: Universal image-text representation learning

    Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Universal image-text representation learning. In European conference on computer vision, pp.\ 104--120. Springer, 2020 b

    2020

  3. [11]

    and Shashua, A

    Cohen, N. and Shashua, A. Inductive bias of deep convolutional networks through pooling geometry. arXiv preprint arXiv:1605.06743, 2016

    2016 arXiv

  4. [12]

    Algorithms for learning kernels based on centered alignment

    Cortes, C., Mohri, M., and Rostamizadeh, A. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13: 0 795--828, 2012

    2012

  5. [13]

    On kernel-target alignment

    Cristianini, N., Shawe-Taylor, J., Elisseeff, A., and Kandola, J. On kernel-target alignment. Advances in neural information processing systems, 14, 2001

    2001

  6. [14]

    and Marxer, R

    Cuervo, S. and Marxer, R. Scaling properties of speech language models. arXiv preprint arXiv:2404.00685, 2024

    2024 arXiv

  7. [15]

    DelMonte, D. W. and Kim, T. Anatomy and physiology of the cornea. Journal of Cataract & Refractive Surgery, 37 0 (3): 0 588--598, 2011

    2011

  8. [16]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  9. [17]

    Diamond, J. M. and Ordunio, D. Guns, germs, and steel, volume 521. Books on Tape New York, 1999

    1999

  10. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  11. [19]

    Its not a modality gap: Characterizing and addressing the contrastive gap

    Fahim, A., Murphy, A., and Fyshe, A. Its not a modality gap: Characterizing and addressing the contrastive gap. arXiv preprint arXiv:2405.18570, 2024

    2024 arXiv

  12. [20]

    Eva: Exploring the limits of masked visual representation learning at scale

    Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 19358--19369, 2023

    2023

  13. [21]

    Learning robust representations via multi-view information bottleneck

    Federici, M., Dutta, A., Forr \'e , P., Kushman, N., and Akata, Z. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017, 2020

    2002 arXiv

  14. [22]

    Towards artificial general intelligence via a multimodal foundation model

    Fei, N., Lu, Z., Gao, Y., Yang, G., Huo, Y., Wen, J., Lu, H., Song, R., Gao, X., Xiang, T., et al. Towards artificial general intelligence via a multimodal foundation model. Nature Communications, 13 0 (1): 0 3094, 2022

    2022

  15. [23]

    S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T

    Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26, 2013

    2013

  16. [24]

    V., Joulin, A., and Misra, I

    Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 15180--15190, 2023

    2023

  17. [25]

    W., Wuthrich, M., Miladinovic, D., Locatello, F., Breidt, M., Volchkov, V., Akpo, J., Bachem, O., Sch \"o lkopf, B., and Bauer, S

    Gondal, M. W., Wuthrich, M., Miladinovic, D., Locatello, F., Breidt, M., Volchkov, V., Akpo, J., Bachem, O., Sch \"o lkopf, B., and Bauer, S. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. Advances in Neural Information Proc...

    2019

  18. [26]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  19. [27]

    Data-efficient image recognition with contrastive predictive coding

    Henaff, O. Data-efficient image recognition with contrastive predictive coding. In International conference on machine learning, pp.\ 4182--4192. PMLR, 2020

    2020

  20. [28]

    D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y

    Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018

    2018 arXiv

  21. [29]

    The platonic representation hypothesis

    Huh, M., Cheung, B., Wang, T., and Isola, P. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024

    2024 arXiv

  22. [30]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp.\ 4904--4916. PMLR, 2021

    2021

  23. [31]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  24. [32]

    and Fei-Fei, L

    Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3128--3137, 2015

    2015

  25. [33]

    Similarity of neural network models: A survey of functional and representational measures

    Klabunde, M., Schumacher, T., Strohmaier, M., and Lemmerich, F. Similarity of neural network models: A survey of functional and representational measures. arXiv preprint arXiv:2305.06329, 2023

    2023 arXiv

  26. [34]

    Multimodal machine learning in precision health: A scoping review

    Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., and Luo, Y. Multimodal machine learning in precision health: A scoping review. npj Digital Medicine, 5 0 (1): 0 171, 2022

    2022

  27. [35]

    Similarity of neural network representations revisited

    Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similarity of neural network representations revisited. In International conference on machine learning, pp.\ 3519--3529. PMLR, 2019

    2019

  28. [36]

    Kotas, M. E. and Medzhitov, R. Homeostasis, inflammation, and disease susceptibility. Cell, 160 0 (5): 0 816--827, 2015

    2015

  29. [37]

    Multimodality of ai for education: Towards artificial general intelligence

    Lee, G.-G., Shi, L., Latif, E., Gao, Y., Bewersdorff, A., Nyaaba, M., Guo, S., Wu, Z., Liu, Z., Wang, H., et al. Multimodality of ai for education: Towards artificial general intelligence. arXiv preprint arXiv:2312.06037, 2023

    2023 arXiv

  30. [38]

    Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34: 0 9694--9705, 2021

    2021

  31. [39]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp.\ 12888--12900. PMLR, 2022

    2022

  32. [40]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023

    2023

  33. [41]

    W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J

    Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35: 0 17612--17625, 2022

    2022

  34. [42]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp....

    2014

  35. [43]

    and Lee, H

    Logeswaran, L. and Lee, H. An efficient framework for learning sentence representations. arXiv preprint arXiv:1803.02893, 2018

    2018 arXiv

  36. [44]

    dsprites: Disentanglement testing sprites dataset

    Matthey, L., Higgins, I., Hassabis, D., and Lerchner, A. dsprites: Disentanglement testing sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017

    2017

  37. [45]

    Mitchell, B. R. The spatial inductive bias of deep learning. PhD thesis, Johns Hopkins University, 2017

    2017

  38. [46]

    Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  39. [47]

    Bleu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

    2002

  40. [48]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  41. [49]

    Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability

    Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017

    2017

  42. [50]

    Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 0 12116--12128, 2021

    Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 0 12116--12128, 2021

    2021

  43. [51]

    Accept the modality gap: An exploration in the hyperbolic space

    Ramasinghe, S., Shevchenko, V., Avraham, G., and Thalaiyasingam, A. Accept the modality gap: An exploration in the hyperbolic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 27263--27272, 2024

    2024

  44. [52]

    Hierarchical text-conditional image generation with clip latents

    Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1 0 (2): 0 3, 2022

    2022 arXiv

  45. [53]

    T., Argus, M., Fischer, V., and Brox, T

    Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., and Brox, T. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning. arXiv preprint arXiv:2404.07983, 2024

    2024 arXiv

  46. [54]

    Facenet: A unified embedding for face recognition and clustering

    Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 815--823, 2015

    2015

  47. [55]

    and Tishby, N

    Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017

    2017 arXiv

  48. [56]

    Human computer interaction

    Sinha, G., Shahi, R., and Shankar, M. Human computer interaction. In 2010 3rd International Conference on Emerging Trends in Engineering and Technology, pp.\ 1--4. IEEE, 2010

    2010

  49. [57]

    Contrastive multiview coding

    Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XI 16, pp.\ 776--794. Springer, 2020 a

    2020

  50. [58]

    What makes for good views for contrastive learning? Advances in neural information processing systems, 33: 0 6827--6839, 2020 b

    Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P. What makes for good views for contrastive learning? Advances in neural information processing systems, 33: 0 6827--6839, 2020 b

    2020

  51. [59]

    and Zaslavsky, N

    Tishby, N. and Zaslavsky, N. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp.\ 1--5. IEEE, 2015

    2015

  52. [60]

    C., and Bialek, W

    Tishby, N., Pereira, F. C., and Bialek, W. The information bottleneck method. arXiv preprint physics/0004057, 2000

    2000 arXiv

  53. [61]

    H., Wu, Y., Salakhutdinov, R., and Morency, L.-P

    Tsai, Y.-H. H., Wu, Y., Salakhutdinov, R., and Morency, L.-P. Self-supervised learning from a multi-view perspective. arXiv preprint arXiv:2006.05576, 2020

    2006 arXiv

  54. [62]

    Understanding and fixing the modality gap in vision-language models

    Udandarao, V. Understanding and fixing the modality gap in vision-language models. Master’s thesis, University of Cambridge, 2022

    2022

  55. [63]

    Ranking with ordered weighted pairwise classification

    Usunier, N., Buffoni, D., and Gallinari, P. Ranking with ordered weighted pairwise classification. In Proceedings of the 26th annual international conference on machine learning, pp.\ 1057--1064, 2009

    2009

  56. [64]

    Cider: Consensus-based image description evaluation

    Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4566--4575, 2015

    2015

  57. [65]

    and Liu, H

    Wang, F. and Liu, H. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 2495--2504, 2021

    2021

  58. [66]

    Rethinking minimal sufficient representation in contrastive learning

    Wang, H., Guo, X., Deng, Z.-H., and Lu, Y. Rethinking minimal sufficient representation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 16041--16050, 2022

    2022

  59. [67]

    and Isola, P

    Wang, T. and Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, pp.\ 9929--9939. PMLR, 2020

    2020

  60. [68]

    Disentangled representation learning

    Wang, X., Chen, H., Wu, Z., Zhu, W., et al. Disentangled representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  61. [69]

    and Wu, L

    Wang, Z. and Wu, L. Theoretical analysis of the inductive biases in deep convolutional networks. Advances in Neural Information Processing Systems, 36: 0 74289--74338, 2023

    2023

  62. [70]

    Large scale image annotation: learning to rank with joint word-image embeddings

    Weston, J., Bengio, S., and Usunier, N. Large scale image annotation: learning to rank with joint word-image embeddings. Machine learning, 81: 0 21--35, 2010

    2010

  63. [71]

    X., and Lin, D

    Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3733--3742, 2018

    2018

  64. [72]

    Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., and L \'o pez, A. M. Multimodal end-to-end autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 23 0 (1): 0 537--547, 2020

    2020

  65. [73]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Xu, H., Ghosh, G., Huang, P.-Y., Okhonko, D., Aghajanyan, A., Metze, F., Zettlemoyer, L., and Feichtenhofer, C. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021

    2021 arXiv

  66. [74]

    A theory of usable information under computational constraints

    Xu, Y., Zhao, S., Song, J., Stewart, R., and Ermon, S. A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689, 2020

    2002 arXiv

  67. [75]

    Yager, R. R. On ordered weighted averaging aggregation operators in multicriteria decisionmaking. IEEE Transactions on systems, Man, and Cybernetics, 18 0 (1): 0 183--190, 1988

    1988

  68. [76]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.