Pith. sign in

REVIEW 3 major objections 4 minor 75 references

Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper gives an information-theoretic bound on how much a multimodal large language model's performance can degrade under distribution shifts, tying the degradation to visual and textual input divergence plus output drift.

desk verdict Good new metric and plausible bound, but the empirical validation does not test the inequality and the preference-model equivalence is wrong; worth a serious referee, not ready as is. read the letter →

arxiv 2502.00577 v2 pith:FPFRXLS7 submitted 2025-02-01 cs.AI cs.CLcs.LG

Changdae Oh , Zhen Fang , Shawn Im , Xuefeng Du , Yixuan Li This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords effectivemutualinformationdistributionshiftmultimodallargelanguagemodelstheoryout-of-distributiongeneralizationJensen-ShannondivergenceLLM-as-a-judgevisualinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes the first formal, information-theoretic account of why multimodal LLMs lose performance when evaluation data drifts from instruction-tuning data. It introduces effective mutual information (EMI), which subtracts the data's baseline mutual information from the model's query-response mutual information so that domain scale does not confound the measurement. The central result is an upper bound on the EMI difference between in-distribution and out-of-distribution data: the maximum performance gap is controlled by the Jensen-Shannon divergences of the visual and textual input marginals, plus a term measuring how far the model's output distribution is from the true conditional. The paper reports strong correlations between this bound and observed EMI gaps across 61 shift scenarios, suggesting the bound is practically predictive.

What carries the argument

The load-bearing object is effective mutual information (EMI), defined as $\mathrm{EMI}(P_{XY}; P_\theta) = I(P_X \otimes P_\theta) - I(P_{XY})$, which removes the domain-dependent baseline mutual information from the model's query-response mutual information. The paper then forms EMID as the difference between EMI on the in-distribution and out-of-distribution joint distributions, and bounds EMID by decomposing it into entropy differences. The proof machinery uses Jensen-Shannon divergences and a sub-Gaussian inequality to convert conditional-entropy differences into divergence terms, producing the simplified bound $b_H(\sqrt{D_{\mathrm{JS}}(P_{X_v}\|Q_{X_v})} + \sqrt{D_{\mathrm{JS}}(P_{X_t}\|Q_{X_t})}) + 8\Delta^{1/4}$ and a looser general bound that also includes conditional-dependency and $Y|X$ discrepancy terms.

What would settle it

Find one ID-OOD pair satisfying the consistency-of-conditionals assumption whose EMID, estimated by a validated mutual-information estimator, exceeds $b_H(\sqrt{D_{\mathrm{JS}}(P_{X_v}\|Q_{X_v})} + \sqrt{D_{\mathrm{JS}}(P_{X_t}\|Q_{X_t})}) + 8\Delta^{1/4}$ computed with reliable JS estimates; that single violation would disprove the paper's simplified bound. In the paper's own protocol, this means checking the unreplaced bound rather than the scale-adjusted proxy with the constant 4.0 and the model's conditional in place of the true one.

Watch

Extended reading notes

Core claim

The central discovery is that the maximum performance gap of an MLLM under distribution shift can be characterized by a few measurable divergences. Defining $\mathrm{EMID}(P_{XY}, Q_{XY}; P_\theta) = \mathrm{EMI}(P_{XY}; P_\theta) - \mathrm{EMI}(Q_{XY}; P_\theta)$, the paper proves that when the conditional structure is stable across ID and OOD, EMID is at most $b_H(\sqrt{D_{\mathrm{JS}}(P_{X_v}\|Q_{X_v})} + \sqrt{D_{\mathrm{JS}}(P_{X_t}\|Q_{X_t})}) + 8\Delta^{1/4}$, where $b_H$ is a bounded entropy scale and $\Delta$ collects JS divergences between the model's marginal output distribution and the true output distribution on both domains. In the general case without conditional consistency, the bound additionally includes discrepancies of the conditional dependencies and of $Y|X$. The practical content is that visual and textual distributional discrepancies add, joint shifts hurt more than either alone, and the inequality can be used to estimate, regularize, and control the degradation.

Load-bearing premise

The simplified bound assumes that the conditional distributions (image given text, text given image, and response given input) are identical in the in-distribution and out-of-distribution settings, and the empirical verification separately assumes that the hand-set scale-adjusted estimate with the model standing in for the true conditional is the bound itself; if either assumption fails, the observed correlations cannot be read as confirming Theorem 4.5.

Editorial extensions

If this is right

  • EMI can substitute for expensive LLM-judge scoring: the paper shows it correlates with relative preference scores across four MLLMs and is about 138 times faster per dataset.
  • The bound says visual and textual shift severity add, so a joint shift must cause a larger maximum performance gap than either modality shifted alone, matching the observed complementary effect.
  • Because the bound is model-dependent through the output-discrepancy term, different MLLMs can have different predicted sensitivity to the same shift, which the fitted regression slopes in the paper confirm.
  • The upper bound can be turned into a training regularizer that improves shifted-scenario scores while keeping in-distribution scores unchanged in the reported setup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper verifies the bound by correlation rather than by checking each EMID value against the bound, a natural next step is to test the inequality directly on individual ID-OOD pairs and measure how often it is violated; this would turn the current trend evidence into pointwise evidence.
  • The bound's divergences are computed in fixed CLIP and RoBERTa embedding spaces, so the measured “shift” is embedding-dependent; one could ask whether the bound remains predictive under learned joint embeddings, which the paper's E5-V ablation only partially addresses.
  • The framework should transfer to text-only LLMs by dropping the visual term, and the paper hints at this possibility but does not test it.
  • The regularizer's use of an isotropic Gaussian as a stand-in for the unknown OOD distribution is a design choice; testing it against actual target shifts with known conditionals would clarify how much of the robustness gain comes from the bound versus the prior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Effective Mutual Information (EMI) as a relevance metric for multimodal LLM evaluation, defines the EMI difference (EMID) between in-distribution and out-of-distribution data, and derives two upper bounds: Theorem 4.5 for a 'simplified scenario' with consistent conditional distributions and Theorem 4.6 for a general scenario with arbitrary shifts. It also proves a connection between EMI and the LLM-as-a-judge relative preference score (Lemma 4.3 and Theorem 4.4). The empirical sections report correlations between EMI and RP score, between EMID and a scale-adjusted proxy of the simplified bound, across 34 synthetic and 27 natural shift scenarios with four LLaVA models and two additional MLLMs, and an instruction-tuning regularization based on the bound.

Significance. If Theorem 4.6 is correct, the paper provides a genuinely useful first step: a formal decomposition of an information-theoretic performance gap into visual and textual marginal discrepancies plus conditional and output terms, together with a formal connection to the RP score. The empirical effort is a real strength: 61 shift scenarios, six MLLMs, and ablation over multiple MI estimators, JSD estimators, and embedding spaces (Tables 2-5 and 7-8) give the paper a broad experimental base, and the proposed EMI-based regularization is an interesting application. The derivation is not circular in the sense that the bound follows from stated assumptions. However, the headline 'simplified scenario' theorem is vacuous under its stated assumptions, and the reported experiments do not test the inequality in the bound. The contribution is therefore currently an interesting but unverified theoretical edifice rather than a validated framework.

major comments (3)
  1. [§4.3 / Appendix D.2 (Theorem 4.5, Assumption D.10)] Theorem 4.5 is vacuous in the setting it names. Assumption D.10 requires P_Xt|Xv = Q_Xt|Xv and P_Xv|Xt = Q_Xv|Xt. These two equalities together force P_Xv = Q_Xv and P_Xt = Q_Xt (indeed PXY = QXY), because the joint ratio P(Xv,Xt)/Q(Xv,Xt) is simultaneously expressible as a function of Xt alone and of Xv alone. Hence the two Jensen-Shannon terms in Eq. (10) are identically zero, and with P_Y|X = Q_Y|X the EMID itself is zero. The theorem therefore bounds only the null-shift case; it does not characterize visual-only, text-only, or joint shifts as claimed in the 'Implication' paragraph of §4.3, and those special cases in fact violate Assumption D.10. Section 5's 61 shift scenarios are defined by changes in marginals or joint structure, so they cannot serve as evidence for Theorem 4.5.
  2. [§5 and Appendix B] The 'Verification of bound' experiment does not test the inequality in Theorem 4.5. Appendix B constructs the quantity that is correlated with EMID by setting bH = 4.0 from pilot observations, replacing Q_Y|X with P_theta, and then applying a further rescaling; the resulting 'scale-adjusted upper bound' is a heuristic proxy, not the theorem's RHS. Pearson correlation in Table 3 only measures co-movement across scenarios; it is compatible with EMID exceeding the bound on every scenario and says nothing about whether the pointwise inequality holds. A direct validation should report per-scenario bound values, slack or ratios, and the count of violations, or should be explicitly framed as an exploratory correlation study rather than as a verification of the theorem.
  3. [§4.3 and §5] Because Theorem 4.5 is null-shift empty, the paper's only non-vacuous formal statement about shifts is Theorem 4.6, yet Section 4.3 says 'we focus on the validation of Theorem 4.5 in the following section,' and Section 5 never estimates the conditional-divergence terms (the Dbar_JS terms) or the output-conditional term of Theorem 4.6. As a result, the empirical support for the advertised 'first theoretical framework' is currently missing for the theorem that actually applies to the 61 tested shift scenarios. The general theorem should either be validated, or the contribution should be narrowed accordingly.
minor comments (4)
  1. [Appendix C.2] The sentence reporting the Pearson correlation refers to 'that of Theorem 10'; this should be a reference to Eq. (10) or to Theorem 4.5.
  2. [Section 3] 'more results from different types of shifts can be founded in Appendix C' should read 'found in Appendix C'.
  3. [Figure 3] The axis labels should state explicitly that the right-hand quantity is the Appendix B scale-adjusted proxy, not the bound from Eq. (10).
  4. [Appendix B, Eq. (14)] The symbol H is used both for the batch entropy estimate and for the entropy scale bH in the theorem; using a different symbol for the batch estimate would reduce ambiguity.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the EMID bound is derived from definitions and standard inequalities; the empirical scale-adjusted proxy is a disclosed approximation, not a fitted prediction.

full rationale

The central derivation is self-contained. EMI and EMID are defined in Definitions 4.1/4.2 and Eq. (9), and Theorems 4.5/4.6 are proven in Appendix D by expanding EMID into entropy differences (Eq. (D.2)) and applying Lemma D.4, Lemma D.9, and a conditional-decomposition identity; no fitted parameter enters the theorem statements themselves. The empirical validation in Section 5 and Appendix B does use a 'scale-adjusted' bound with bH set to 4.0 from pilot observations and with Q_{Y|X} replaced by P_theta, but the authors explicitly state 'it is not the exact estimate of EMID UB' and note that Pearson correlation is invariant to the linear scale; this is a disclosed approximation that weakens the empirical test, not a circular reduction of EMID to the bound. The consistency assumption of Theorem 4.5 does not by itself force the marginal JS terms to vanish (e.g., identity-coupling joint distributions satisfy the equal conditionals while having different marginals), so the simplified bound is not vacuous by construction. No load-bearing self-citation or imported uniqueness theorem is invoked. The claimed special cases of visual-only and text-only shifts are not fully compatible with the consistency assumption, which is a correctness concern rather than a circularity. Score 2 reflects the minor in-sample and approximation caveats, not circular reasoning.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central theorem relies on standard information-theoretic inequalities (Pinsker, sub-Gaussian concentration), a stated representational-capacity assumption, and a consistency-of-conditionals assumption for the simplified bound. The empirical claims additionally assume that embeddings from CLIP/XLM-RoBERTa and the CLUB/RJSD estimators faithfully represent the true divergences. The only hand-fitted scalar is the bH proxy set to 4.0 in Appendix B.

free parameters (1)
  • bH proxy (entropy scale) = 4.0 (approximated from pilot H~ centered about 2.0)
    Appendix B replaces bH = max_x[H(Q_Y|X=x)+H(P_theta(·|x))] with a constant 4.0 to construct the scale-adjusted upper bound. While scale-invariant to Pearson correlation, this is a hand-chosen value, so the reported bound is not the literal theorem bound.
assumptions (5)
  • domain assumption epsilon-representation capacity: min_theta E_PX D_KL(P_Y|X=x || P_theta(·|x)) <= epsilon
    Used in Theorem 4.4 (Eq. 8) to guarantee the optimally tuned model is close to the true conditional, making the EMI-PM gap small.
  • domain assumption PXY > c > 0 for some constant c
    Assumed for Theorem 4.4; authors argue it holds because PXY is strictly positive on a compact space. Used in Lemma D.7.
  • ad hoc to paper Consistency of conditional distributions (Assumption D.10)
    Required for Theorem 4.5; sets conditional divergences to zero. Real shifts often change conditional dependencies, so this is a restrictive but explicit premise.
  • standard math Sub-Gaussian boundedness of f(x) in Lemma D.9 (imported from Shui et al.)
    Needed for the log-moment bound used to control the difference in conditional entropy expectations under P and Q.
  • domain assumption Frozen embedding spaces (CLIP, XLM-RoBERTa) capture the true distributional discrepancies
    All empirical estimates of MI and JS divergences are computed in these embedding spaces; the validity of the empirical validation depends on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach." pith.science (2026). https://pith.science/paper/FPFRXLS7

@misc{pith2026250200577,
  author       = {Pith},
  title        = {Pith review of: Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FPFRXLS7}},
  note         = {Machine review of arXiv:2502.00577}
}
read the original abstract

Multimodal large language models (MLLMs) have shown promising capabilities but struggle under distribution shifts, where evaluation data differ from instruction tuning distributions. Although previous works have provided empirical evaluations, we argue that establishing a formal framework that can characterize and quantify the risk of MLLMs is necessary to ensure the safe and reliable application of MLLMs in the real world. By taking an information-theoretic perspective, we propose the first theoretical framework that enables the quantification of the maximum risk of MLLMs under distribution shifts. Central to our framework is the introduction of Effective Mutual Information (EMI), a principled metric that quantifies the relevance between input queries and model responses. We derive an upper bound for the EMI difference between in-distribution (ID) and out-of-distribution (OOD) data, connecting it to visual and textual distributional discrepancies. Extensive experiments on real benchmark datasets, spanning 61 shift scenarios, empirically validate our theoretical insights.

Figures

Figures reproduced from arXiv: 2502.00577 by the authors.

Figure 1
Figure 1. Performance variation against varying distribution shifts. We evaluated LLaVA v1.5 (top) and LLaVA NeXT (bottom) models on 27 out-of-distribution (OOD) variants of the LLaVA-Bench COCO (ID). Here, the x-axis is sorted by the severity of shifts between ID and OOD. There is a consistent trend, increased degrees of distribution shifts result in performance degradations of MLLM [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Types of distribution shifts between train and evalu￾ation of MLLMs. We simulate visual, text, and joint shifts by controlling the shift of each input modality. 1. Visual shift: the marginal distribution of visual query undergoes shift D(PXv ∥QXv ) ≫ 0, while that of text query remains largely unchanged D(PXt ∥QXt ) ≈ 0. 2. Text shift: the marginal distribution of text query un￾dergoes shift D(PXt ∥QXt ) ≫ 0, while … view at source ↗
Figure 3
Figure 3. Scatter plot with regression line between empirical estimates of EMID and its upper bound. Over the 34 synthetic and 27 natural distribution shift scenarios, we evaluate four MLLMs and get 136 cases and 108 cases of synthetic shifts and natural shifts, respectively, for visualizing EMID and its scale-adjusted upper bound estimates (See Appendix B for details). The two panels on the left show results for all four mod… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Scatter plot with regression line between empirical estimates of EMID and partial components of its upper bound. We remove the ∆ term of bound (Eq. (10)) and only use the estimates of JSD terms over visual and text inputs. Validation with advanced MLLMs. So far, our ev…
Figure 5
Figure 5. Figure 5: Information diagram and motivation of effective mutual information. The difference between vanilla MI terms does not consider the domain-dependent intrinsic scale and mutual information, thereby failing to fairly measure the relevance between input query x and model pr…
Figure 6
Figure 6. Figure 6: Performance variation against varying degrees of distribution shifts. We evaluated LLaVA v1.5 and LLaVA NeXT models on 34 out-of-distribution (OOD) variants induced by image and text perturbations of the LLaVA-Bench COCO dataset (ID). Here, the x-axis is sorted by the …
Figure 7
Figure 7. Figure 7: Sensitivity analysis for batch size, learning rate, and hidden dimension during MI estimator training. For the considered hyperparameter searching grid, MI estimates derived by the CLUB estimator robustly achieve high Spearman correlation with RP score (the largest dev…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 51 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    A., Fischer, I., Dillon, J

    Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016

    arXiv 2016

  4. [4]

    I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D

    Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, pp.\ 531--540. PMLR, 2018

    work page 2018

  5. [5]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  6. [6]

    Bu, Y., Zou, S., Liang, Y., and Veeravalli, V. V. Estimation of kl divergence: Optimal minimax rate. IEEE Transactions on Information Theory, 64 0 (4): 0 2648--2674, 2018

    work page 2018

  7. [7]

    Castro, R. M. and Nowak, R. D. Minimax bounds for active learning. IEEE Transactions on Information Theory, 54 0 (5): 0 2339--2353, 2008

    work page 2008

  8. [8]

    Rev: Information-theoretic evaluation of free-text rationales

    Chen, H., Brahman, F., Ren, X., Ji, Y., Choi, Y., and Swayamdipta, S. Rev: Information-theoretic evaluation of free-text rationales. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 2007--2030, 2023

    work page 2007

Show all 75 references
  1. [9]

    Infogan: Interpretable representation learning by information maximizing generative adversarial nets

    Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016

    2016

  2. [10]

    Club: A contrastive log-ratio upper bound of mutual information

    Cheng, P., Hao, W., Dai, S., Liu, J., Gan, Z., and Carin, L. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pp.\ 1779--1788. PMLR, 2020

    2020

  3. [11]

    Fairfil: Contrastive neural debiasing method for pretrained text encoders

    Cheng, P., Hao, W., Yuan, S., Si, S., and Carin, L. Fairfil: Contrastive neural debiasing method for pretrained text encoders. In International Conference on Learning Representations, 2021

    2021

  4. [12]

    Unsupervised cross-lingual representation learning at scale

    Conneau, A. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019

    1911 arXiv

  5. [13]

    Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: towards general-purpose vision-language models with instruction tuning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp.\ ...

    2023

  6. [14]

    An information-theoretic approach to distribution shifts

    Federici, M., Tomioka, R., and Forr \'e , P. An information-theoretic approach to distribution shifts. Advances in Neural Information Processing Systems, 34: 0 17628--17641, 2021

    2021

  7. [15]

    Fraser, A. M. and Swinney, H. L. Independent coordinates for strange attractors from mutual information. Physical review A, 33 0 (2): 0 1134, 1986

    1986

  8. [16]

    Finetune like you pretrain: Improved finetuning of zero-shot vision models

    Goyal, S., Kumar, A., Garg, S., Kolter, Z., and Raghunathan, A. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 19338--19347, 2023

    2023

  9. [17]

    How well does gpt-4v (ision) adapt to distribution shifts? a preliminary investigation

    Han, Z., Zhou, G., He, R., Wang, J., Wu, T., Yin, Y., Khan, S., Yao, L., Liu, T., and Zhang, K. How well does gpt-4v (ision) adapt to distribution shifts? a preliminary investigation. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024

    2024

  10. [18]

    Clipscore: A reference-free evaluation metric for image captioning

    Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021

    2021

  11. [19]

    Hewitt, J., Ethayarajh, K., Liang, P., and Manning, C. D. Conditional probing: measuring usable information beyond a baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.\ 1626--1639, 2021

    2021

  12. [20]

    Hoyos, J. K. and Giraldo, L. G. S. A kernel two-sample test with the representation jensen-shannon divergence. In Latinx in AI @ NeurIPS 2024, 2024. URL https://openreview.net/forum?id=bKZbWy3DnR

    2024

  13. [21]

    Hoyos-Osorio, J. K. and Sanchez-Giraldo, L. G. The representation jensen-shannon divergence. arXiv preprint arXiv:2305.16446, 2023

    2023 arXiv

  14. [22]

    Hunter, D. R. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32 0 (1): 0 384--406, 2004

    2004

  15. [23]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  16. [24]

    E5-v: Universal embeddings with multimodal large language models

    Jiang, T., Song, M., Zhang, Z., Huang, H., Deng, W., Sun, F., Zhang, Q., Wang, D., and Zhuang, F. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580, 2024

    2024 arXiv

  17. [25]

    Ju, H., Li, D., and Zhang, H. R. Robust fine-tuning of deep neural networks with hessian-based generalization guarantees. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learni...

    2022

  18. [26]

    Prometheus: Inducing fine-grained evaluation capability in language models

    Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., et al. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  19. [27]

    Rademacher complexities and bounding the excess risk in active learning

    Koltchinskii, V. Rademacher complexities and bounding the excess risk in active learning. The Journal of Machine Learning Research, 11: 0 2457--2485, 2010

    2010

  20. [28]

    A mutual information maximization perspective of language representation learning

    Kong, L., de Masson d'Autume, C., Yu, L., Ling, W., Dai, Z., and Yogatama, D. A mutual information maximization perspective of language representation learning. In International Conference on Learning Representations, 2020

    2020

  21. [29]

    Estimating mutual information

    Kraskov, A., St \"o gbauer, H., and Grassberger, P. Estimating mutual information. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 69 0 (6): 0 066138, 2004

    2004

  22. [30]

    Fine-tuning can distort pretrained features and underperform out-of-distribution

    Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022

    2022

  23. [31]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., and Gao, J. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [32]

    Robust prompt optimization for large language models against distribution shifts

    Li, M., Wang, W., Feng, F., Cao, Y., Zhang, J., and Chua, T.-S. Robust prompt optimization for large language models against distribution shifts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 1539--1554, 2023 a

    2023

  25. [33]

    and Turner, R

    Li, Y. and Turner, R. E. R \'e nyi divergence variational inference. Advances in neural information processing systems, 29, 2016

    2016

  26. [34]

    Evaluating object hallucination in large vision-language models

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 292--305, 2023 b . URL https://aclanthology.org/2023.em...

    2023

  27. [35]

    Aligning instruction tuning with pre-training

    Liang, Y., Zheng, T., Du, X., Zhang, G., Qu, X., Yue, X., Zheng, C., Liu, J., Ma, L., Chen, W., et al. Aligning instruction tuning with pre-training. arXiv preprint arXiv:2501.09368, 2025

    2025 arXiv

  28. [36]

    Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A., and Sutherland, D. J. Learning deep kernels for non-parametric two-sample tests. In International conference on machine learning, pp.\ 6316--6326. PMLR, 2020

    2020

  29. [37]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Conference on Neural Information Processing Systems (NeurIPS), 36, 2023

    2023

  30. [38]

    Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26296--26306, 2024 a

    2024

  31. [39]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 b

    2024

  32. [40]

    Foundations of machine learning, 2018

    Mohri, M. Foundations of machine learning, 2018

    2018

  33. [41]

    J., and Jordan, M

    Nguyen, X., Wainwright, M. J., and Jordan, M. I. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56 0 (11): 0 5847--5861, 2010

    2010

  34. [42]

    Towards calibrated robust fine-tuning of vision-language models

    Oh, C., Kim, M., Lim, H., Park, J., Jeong, E., Cheng, Z.-Q., and Song, K. Towards calibrated robust fine-tuning of vision-language models. Advances in Neural Information Processing Systems, 37, 2025 a

    2025

  35. [43]

    Dawin: Training-free dynamic weight interpolation for robust adaptation

    Oh, C., Li, Y., Song, K., Yun, S., and Han, D. Dawin: Training-free dynamic weight interpolation for robust adaptation. In The Thirteenth International Conference on Learning Representations, 2025 b

    2025

  36. [44]

    Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  37. [45]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  38. [46]

    Estimation of entropy and mutual information

    Paninski, L. Estimation of entropy and mutual information. Neural computation, 15 0 (6): 0 1191--1253, 2003

    2003

  39. [47]

    Pinsker, M. S. Information and information stability of random variables and processes. Holden-Day, 1964

    1964

  40. [48]

    On variational bounds of mutual information

    Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. On variational bounds of mutual information. In International Conference on Machine Learning, pp.\ 5171--5180. PMLR, 2019

    2019

  41. [49]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  42. [50]

    Shannon, C. E. A mathematical theory of communication. The Bell system technical journal, 27 0 (3): 0 379--423, 1948

    1948

  43. [51]

    A novel domain adaptation theory with jensen--shannon divergence

    Shui, C., Chen, Q., Wen, J., Zhou, F., Gagn \'e , C., and Wang, B. A novel domain adaptation theory with jensen--shannon divergence. Knowledge-Based Systems, 257: 0 109808, 2022

    2022

  44. [52]

    and Tishby, N

    Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017

    2017 arXiv

  45. [53]

    Simon, H. A. Spurious correlation: A causal interpretation. Journal of the American Statistical Association, 49 0 (267): 0 467--479, 1954

    1954

  46. [54]

    and Rawat, A

    Sinn, M. and Rawat, A. Non-parametric estimation of jensen-shannon divergence in generative adversarial network training. In International Conference on Artificial Intelligence and Statistics, pp.\ 642--651. PMLR, 2018

    2018

  47. [55]

    An information-theoretic approach to prompt engineering without ground truth labels

    Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., and Wingate, D. An information-theoretic approach to prompt engineering without ground truth labels. In Proceedings of the 60th Annual Meeting of the Association for Computationa...

    2022

  48. [56]

    and Goldfeld, Z

    Sreekumar, S. and Goldfeld, Z. Neural estimation of statistical divergences. Journal of machine learning research, 23 0 (126): 0 1--75, 2022

    2022

  49. [57]

    K., Fukumizu, K., Gretton, A., Sch \"o lkopf, B., and Lanckriet, G

    Sriperumbudur, B. K., Fukumizu, K., Gretton, A., Sch \"o lkopf, B., and Lanckriet, G. R. G. On the empirical estimation of integral probability metrics . Electronic Journal of Statistics, 6: 0 1550 -- 1599, 2012

    2012

  50. [58]

    Measuring robustness to natural distribution shifts in image classification

    Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33: 0 18583--18599, 2020

    2020

  51. [59]

    K., Khetan, A., Lin, Z., and Oh, S

    Thekumparampil, K. K., Khetan, A., Lin, Z., and Oh, S. Robustness of conditional gans to noisy labels. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associ...

    2018

  52. [60]

    Trainable projected gradient method for robust fine-tuning

    Tian, J., He, Z., Dai, X., Ma, C.-Y., Liu, Y.-C., and Kira, Z. Trainable projected gradient method for robust fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7836--7845, 2023

    2023

  53. [61]

    K., Gelly, S., and Lucic, M

    Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lucic, M. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2020

    2020

  54. [62]

    A., Saeidi, A., Hegde, S., Therala, A., Bardoliya, F

    Verma, A. A., Saeidi, A., Hegde, S., Therala, A., Bardoliya, F. D., Machavarapu, N., Ravindhiran, S. A. K., Malyala, S., Chatterjee, A., Yang, Y., et al. Evaluating multimodal large language models across distribution shifts and augmentations. In Proceedings of the IEEE/CVF Co...

    2024

  55. [63]

    Wainwright, M. J. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019

    2019

  56. [64]

    Info \ bert \ : Improving robustness of language models from an information theoretic perspective

    Wang, B., Wang, S., Cheng, Y., Gan, Z., Jia, R., Li, B., and Liu, J. Info \ bert \ : Improving robustness of language models from an information theoretic perspective. In International Conference on Learning Representations, 2021

    2021

  57. [65]

    W., Li, M., Kornblith, S., Roelofs, R., Lopes, R

    Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 7959--7...

    2022

  58. [66]

    and Barron, A

    Yang, Y. and Barron, A. Information-theoretic determination of minimax rates of convergence. Annals of Statistics, pp.\ 1564--1599, 1999

    1999

  59. [67]

    Beaf: Observing before-after changes to evaluate hallucination in vision-language models

    Ye-Bin, M., Hyeon-Woo, N., Choi, W., and Oh, T.-H. Beaf: Observing before-after changes to evaluate hallucination in vision-language models. In European Conference on Computer Vision, pp.\ 232--248. Springer, 2025

    2025

  60. [68]

    Bartscore: Evaluating generated text as text generation

    Yuan, W., Neubig, G., and Liu, P. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34: 0 27263--27277, 2021

    2021

  61. [69]

    J., and Ma, Y

    Zhai, Y., Tong, S., Li, X., Cai, M., Qu, Q., Lee, Y. J., and Ma, Y. Investigating the catastrophic forgetting in multimodal large language model fine-tuning. In Conference on Parsimony and Learning (Proceedings Track), 2024

    2024

  62. [70]

    Q., and Artzi, Y

    Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020

    2020

  63. [71]

    On the out-of-distribution generalization of multimodal large language models

    Zhang, X., Li, J., Chu, W., Hai, J., Xu, R., Yang, Y., Guan, S., Xu, J., and Cui, P. On the out-of-distribution generalization of multimodal large language models. arXiv preprint arXiv:2402.06599, 2024 a

    2024 arXiv

  64. [72]

    Why are visually-grounded language models bad at image classification? Conference on Neural Information Processing Systems (NeurIPS), 2024 b

    Zhang, Y., Unell, A., Wang, X., Ghosh, D., Su, Y., Schmidt, L., and Yeung-Levy, S. Why are visually-grounded language models bad at image classification? Conference on Neural Information Processing Systems (NeurIPS), 2024 b

    2024

  65. [73]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

  66. [74]

    Adapting large multimodal models to distribution shifts: The role of in-context learning

    Zhou, G., Han, Z., Chen, S., Huang, B., Zhu, L., Khan, S., Gao, X., and Yao, L. Adapting large multimodal models to distribution shifts: The role of in-context learning. arXiv preprint arXiv:2405.12217, 2024

    2024 arXiv

  67. [75]

    Mini GPT -4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Mini GPT -4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.