Pith. sign in

REVIEW 4 major objections 4 minor 43 references

The paper establishes the first information-theoretic rate-distortion limit for multimodal retrieval, deriving a single-letter bound on bits-per-query for any ranking system and proving that an entropy-adaptive stochastic quantizer reaches

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:11 UTC pith:3WYK3NRU

load-bearing objection Central converse is false — the chain-rule step reverses the inequality, and a simple two-modality counterexample kills the claimed skew-penalty bound. the 4 major comments →

arxiv 2509.11054 v1 pith:3WYK3NRU submitted 2025-09-14 cs.IT cs.CVmath.IT

Rate-Distortion Limits for Multimodal Retrieval: Theory, Optimal Codes, and Finite-Sample Guarantees

Thomas Y. Chen This is my paper
classification cs.IT cs.CVmath.IT MSC 94A1794A34
keywords rate-distortion theorymultimodal retrievalreciprocal rankmutual informationBlahut-Arimoto algorithmentropy-adaptive temperaturefinite-sample guaranteescontrastive learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that multimodal retrieval has a fundamental information-theoretic limit: how many bits a query must carry to achieve a given ranking quality, measured by reciprocal rank. It derives a single-letter rate-distortion function R(D) with a converse bound that splits into a modality-balanced term plus a skew penalty κΔH, and constructs an explicit entropy-weighted stochastic quantizer with adaptive per-modality temperature that reaches the bound within O(1/n) using n training triples. If true, this answers a basic question—minimum bits per query for any retrieval system—and justifies entropy-aware temperature schedules in practice.

Core claim

The central claim is that for reciprocal-rank distortion, the rate-distortion function for M-modal queries satisfies I(X;C) ≥ R_bal(D) + κΔH (Theorem 4.1), where R_bal is the balanced-source Fano bound, ΔH is the total deviation of marginal entropies from their mean, and κ = (1-ϱ)/(M-1) with ϱ the cross-modal redundancy ratio. The paper further claims this bound is achieved within O(n^{-1}) by an entropy-weighted product quantizer whose decoder temperature is the Blahut-Arimoto fixed point τ*_m ∝ sqrt(Σ_j H_j / (M H_m)). Alongside these, a VC-style bound shows the finite-sample excess distortion scales sub-linearly in both M and ΔH.

What carries the argument

The modality-skew coefficient κ=(1-ϱ)/(M-1), with ϱ=Icross/ΣH_m, which quantifies how entropy imbalance and cross-modal redundancy inflate the required rate; the entropy-weighted product quantizer that allocates bits proportionally to empirical marginal entropies; and the adaptive temperature decoder τ*_m=sqrt(Σ_j Ĥ_j/(M Ĥ_m)), derived as a Blahut-Arimoto fixed point, that turns the rate-distortion optimization into a constructive code.

Load-bearing premise

The converse depends on the unproved premise that a balanced surrogate source with equalized marginal entropies can simulate any encoder for the true source, so that R(D) ≥ R_bal(D), and on the particular chain-rule inequality sum_m H(X_m|C,X_<m) ≥ M H_bal − (1−κ)ΔH; that inequality fails for perfectly redundant modalities.

What would settle it

Take two identical Bernoulli(1/2) modalities, so M=2, H=1, ΔH=0, and κ=0. The proof's inequality claims H(X1|C)+H(X2|C,X1) ≥ 2H=2, but the second term is zero and H(X1|C)≤1, so the asserted lower bound is impossible; consequently the skew-penalty bound cannot follow from the chain rule for this source.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, any retrieval system—contrastive or otherwise—must spend at least R_bal(D)+κΔH bits per query to achieve ranking distortion D; below that, perfect ranking is impossible.
  • Fixed-temperature InfoNCE objectives are information-theoretically suboptimal whenever κ>0, and the gap is exactly κΔH in the rate-distortion trade-off.
  • Entropy-adaptive temperature schedules, rather than a single learned temperature, are justified as the unique Blahut-Arimoto fixed point; using per-modality temperature proportional to 1/sqrt(H_m) is near-optimal.
  • The finite-sample excess risk bound grows only as sqrt((M+ΔH)/n), so adding modalities or imbalance does not explode sample complexity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The bound suggests a practical diagnostic: measure ΔH and ϱ on a corpus, then compute κΔH to tell how many bits are being wasted by imbalance before investing in more parameters or more data.
  • If the skew penalty is real, then contrastive pretraining could be improved by entropy-equalizing transforms (e.g., per-modality whitening or histogram equalization) rather than only tuning temperature; this is a testable prediction the paper does not state.
  • The O(1/n) achievability depends on a Taylor-argument in the supplement that assumes the quantization-cell shift term is one-sided; a careful reader might check whether the cancellation of the linear term holds for non-smooth loss functions, since reciprocal rank is not smooth.
  • Extending the model to RAG, the rate-distortion view might predict that retrieval error (miss rate) and generation error add rather than multiply, matching the paper's speculative hint.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper claims to establish the first information-theoretic rate-distortion limits for multimodal retrieval under reciprocal-rank distortion. The central result, Theorem 4.1, states a converse bound I(X;C) ≥ R_bal(D) + κΔH, where R_bal is a ranking Fano bound and κΔH is a modality-skew penalty. The paper then proposes an entropy-weighted stochastic quantizer with an adaptive per-modality temperature decoder (Theorem 5.1) claimed to achieve the bound to within O(n^{-1}) in finite samples, and a VC/Rademacher-style generalization bound (Theorem 6.2) whose complexity is sub-linear in both modality count M and entropy imbalance ΔH. Experiments on synthetic Gaussian mixtures and Flickr30k are presented as validation.

Significance. The question addressed is timely and practically relevant: understanding the minimum bits-per-query needed for a given reciprocal-rank distortion could inform contrastive retrieval design. The paper also contains a concrete quantization scheme and empirical comparisons, and the adaptive-temperature heuristic is a reasonable engineering suggestion. However, the significance of the theoretical contribution depends entirely on the converse Theorem 4.1. That theorem is not merely under-proved; it is contradicted by a simple two-modality fully-redundant example. Because the lower bound is the foundation for the achievability result, the skew-penalty decomposition, the contrastive-objective implications, and the claimed finite-sample optimality, the paper's central theoretical claims collapse. The experimental results may still be of interest as heuristics, but they do not establish the claimed rate-distortion limits.

major comments (4)
  1. [Sec. 4.1, Eq. (4) and Eq. (5)] The Jensen step has the wrong direction. For rank K>1, convexity of 1/k gives E[1/K | K>1] ≥ 1/E[K | K>1] = 2/(N+2), not 1/(N-1). Thus Eq. (4), p_S ≤ 2(1-D)(N-1)/(N-3), does not follow. Moreover, the subsequent Fano step is not a valid application of Fano's inequality: Fano bounds H(Y|C) in terms of the error probability p_e=1-p_S, but the proof substitutes the expected distortion D for p_e without justification. The inequality R_rank(D) in Eq. (5) is therefore not established. This is load-bearing: Theorem 4.1 is built on R_bal(D), which is defined through R_rank(D). The failure is not a minor gap; a concrete counterexample with M=2, X_1=X_2∼Bernoulli(1/2), N=2, and a decoder returning a uniform permutation gives D=0.25, I(X;C)=0, ΔH=0, ρ=0.5, κ=0.5, and R_bal(0.25)=R_rank(0.25)=log 2 - h(0.25)≈0.189, violating the claimed lower bound (7).
  2. [Sec. 4.2, proof of Theorem 4.1] The proof contains a reversed inequality in the chain-rule step and an unsupported lower bound on the conditional-entropy sum. The chain rule gives I(X;C) = Σ_m [H(X_m|X_<m) - H(X_m|C,X_<m)]. Since H(X_m|X_<m) ≤ H(X_m), replacing the conditional entropy by the marginal entropy H_m yields an upper bound, not the claimed lower bound. The asserted bound Σ_m H(X_m|C,X_<m) ≥ M H_bal - (1-κ)ΔH is stated without derivation; convexity of conditional entropy does not imply it. The sentence 'Bounding each term by conditional entropy and summing' is therefore not a proof. In addition, the balanced-source comparison R(D) ≥ R_bal(D) is asserted with the claim that any encoder for X can be simulated on the balanced source X~; no rigorous argument is given, and it is not a standard rate-distortion identity. Equalizing marginal entropies changes the source distribution, and a code for one source need no
  3. [Sec. 5, Theorem 5.1 and Lemma 5.2] The achievability result does not follow from the stated steps. Step 1 asserts that high-resolution product quantization yields E[||X_m-C_m||^2] = O(2^{-2R_m/d_m}) and then 'hence' E[d] = D*(R)+O(2^{-R_min}); no argument connects Euclidean cell error to the permutation-based reciprocal-rank distortion d((X,Y), g(C)). Step 2 states that the BA fixed point (10) satisfies KKT and therefore minimizes I for the attained distortion, but Eq. (10) is simply written down; it is not derived from the Blahut-Arimoto equations for the ranking distortion. Step 3 bounds the transfer error by O(n^{-1/2}). Lemma 5.2's proof similarly concludes with an O(n^{-1/2}) bound and then asserts that a one-sided 'cell shrink' term cancels in expectation, which is not valid: one-sidedness does not imply zero mean. Thus the claimed O(n^{-1}) finite-sample gap is unsupported. Even if the earlier converse were correct
  4. [Sec. 6, Lemma 6.3] The Gaussian volume approximation and the resulting log-sum inequality are questionable, and the signs appear inconsistent. If the cell probability is written as 2^{-H_m} e^{-τ_m^2}, then the number of cells needed to cover the probability simplex is K_m ≈ 1/(2^{-H_m} e^{-τ_m^2}) = 2^{H_m} e^{τ_m^2}, so log_2 K_m = H_m + τ_m^2/log 2, not H_m - τ_m^2 as written in the proof. With the corrected sign, the claimed inequality Σ_m log K_m ≤ M H - γ^2 ΔH does not follow from the stated approximation. Since the 'graceful scaling' and the sub-linear dependence on ΔH in Theorem 6.2 are obtained through this lemma, that part of the finite-sample analysis is also unsupported.
minor comments (4)
  1. [Sec. 4.1, Eq. (4)] Eq. (4) has a denominator N-3, which is zero at N=3 and negative at N=2. No restriction to N>3 is stated, yet the counterexample in the report uses N=2 and the theorem is claimed for all N.
  2. [Abstract and Sec. 1] The abstract claims O(n^{-1}) while the introduction and Sec. 5.3 also mention O(n^{-1/2}); Lemma 5.2's proof ends with an O(n^{-1/2}) term before claiming cancellation. The rate claims should be made consistent, or the mismatch acknowledged.
  3. [Sec. 5.2, Eq. (10)] The adaptive temperature formula τ_m^* = sqrt(Σ_j H_j / (M H_m)) is presented as a BA fixed point, but no derivation from the BA update is given. A reader cannot verify the claim without additional steps.
  4. [Sec. 7] The paper reports standard errors below 0.6% but omits error bars in Tables 1 and 2. Given the theory's central claim, reporting error bars or confidence intervals would help assess whether the 'within two percentage points' statement is robust.

Circularity Check

1 steps flagged

Theorem 4.1's skew penalty is inserted by definition (Eq. 6) rather than derived; the converse is a definitional artifact, and the claimed chain-rule step is reversed.

specific steps
  1. self definitional [Sec. 4.2, proof of Theorem 4.1 (Eq. 7)]
    "The second sum is lower-bounded by M H_bal − (1−κ)ΔH using convexity of conditional entropy and the definition (6), giving I(X;C)≥R_bal(D)+κΔH."

    κ enters only through its definition (6). The proof asserts an unproved lower bound on Σ_m H(X_m|C,X_<m) and says it holds 'using the definition (6)', so subtracting that bound from Σ_m H_m leaves exactly κΔH: the theorem's skew term is manufactured from the definition, not derived from the source model. The chain-rule direction is also reversed: I(X_m;C|X_<m)=H(X_m|X_<m)−H(X_m|C,X_<m) ≤ H_m−H(X_m|C,X_<m), so the display cannot yield a lower bound on I. A fully redundant M=2 example with X1=X2, N=2, D=0.25 gives I(X;C)=0 but R_bal+κΔH≈0.189, violating (7). Thus the claimed limit is an algebraic artifact of (6) rather than an external bound.

full rationale

The central converse, Theorem 4.1, is the load-bearing result of the paper. Its proof introduces the skew term by defining κ in (6) and then asserts a conditional-entropy inequality 'using the definition (6)' that is not derived from the chain rule, Fano, or any external theorem. No independent benchmark or machine-checked result supplies the κΔH term; it is an input disguised as an output. Because the same formula is reused in Theorem 5.1 and in the empirical 'bound' curves, the claimed price of entropy imbalance reduces to the definition. The Fano bound itself is standard material renamed as the 'ranking Fano bound', but the core circularity is the definitional injection of κΔH in the converse. No self-citation chain is present; the problem is internal definitional circularity in the main limit. Separately, Sec. 6.1 references 'Appendix A for ancillary lemmas', which is absent from the manuscript; this missing support is flagged but does not affect the circularity verdict.

Axiom & Free-Parameter Ledger

4 free parameters · 8 axioms · 3 invented entities

The paper contributes a problem framing, but its central results rest on several asserted inequalities and invented coefficients. Free parameters include the temperature schedule constants gamma and epsilon, the per-modality bit allocation, and the temperature itself; the balanced source and modality-skew coefficient are invented objects without independent support. Standard Fano, Jensen, and Rademacher machinery is used, but in two places the sign of Jensen is wrong.

free parameters (4)
  • per-modality temperature tau_m = tau*_m = sqrt(sum_j Hhat_j / (M Hhat_m)) (Eq 10)
    Introduced as a BA fixed point but inconsistent with the Sec 6.1 schedule; it is a model parameter, not a quantity forced by the data alone.
  • temperature scaling constant gamma = assumed gamma <= 1/sqrt(2) (Sec 6.3)
    Appears in tau_m = gamma sqrt(|H_m - H_bar| + epsilon); chosen by hand, not derived or fit.
  • temperature floor epsilon = small positive, unspecified
    Introduced to avoid zero temperature; no value or sensitivity analysis provided.
  • entropy-based bit allocation R_m = R_m = floor(Hhat_m / sum_j Hhat_j * R)
    A modeling choice for the product quantizer; it is a per-modality allocation of the rate budget, not derived from the converse.
axioms (8)
  • standard math Standard Fano inequality applies to top-1 retrieval with success event S (Sec 4.1)
    The paper invokes Fano's inequality, but its application swaps p_S and 1-p_S and substitutes D incorrectly.
  • ad hoc to paper Jensen's inequality for k -> 1/k bounds E[d] with a specific inequality direction (Sec 4.1)
    The direction used is wrong: for ranks k >= 2, E[1/k] >= (1-p_S)/(N-1), giving E[d] <= ..., not >= ..., so the derived upper bound on p_S is invalid.
  • ad hoc to paper Balanced source X~ exists with equal marginal entropies and every encoder for true X can be simulated on X~ (Sec 4.2)
    The distributions differ, so simulation is not justified; the premise is load-bearing for R(D) >= R_bal(D).
  • ad hoc to paper The conditional-entropy sum satisfies sum_m H(X_m|C,X_<m) >= M H_bal - (1-kappa) Delta H (Sec 4.2 proof)
    Asserted without proof; false for fully redundant modalities, so Theorem 4.1 is unsupported.
  • ad hoc to paper High-resolution product quantization attains E[d] = D*(R) + O(2^{-R_min}) from per-modality MSE rates (Sec 5.3)
    No argument links squared-error quantization to reciprocal-rank distortion; D*(R) is never defined.
  • ad hoc to paper BA fixed point (10) is unique and minimizes the dual objective for ranking distortion (Sec 5.2)
    No BA convergence or KKT verification for the non-additive distortion is provided.
  • ad hoc to paper Gaussian volume approximation gives cell probability 2^{-H_m} e^{-tau_m^2} (Lemma 6.3)
    The formula is not derived and is inconsistent with entropy-constrained quantization.
  • standard math Rademacher complexity and Massart finite-class lemma control uniform deviations (Sec 6.2)
    Standard machinery, invoked with correct general form.
invented entities (3)
  • balanced source X~ no independent evidence
    purpose: Defines R_bal(D) and anchors the converse decomposition (Sec 4.2)
    A hypothetical source with equalized marginal entropies but same joint support; no operational meaning or external verification is provided.
  • modality-skew coefficient kappa no independent evidence
    purpose: Quantifies the claimed entropy-imbalance and redundancy penalty in Thm 4.1
    Defined in Eq (6) as kappa = (1-rho)/(M-1) and used to form the penalty kappa Delta H; the only evidence is the paper's own theorem, which uses the same definition, so no independent falsifiable handle exists.
  • entropy-adaptive temperature schedule no independent evidence
    purpose: Per-modality decoder temperatures claimed to be the BA fixed point and to justify empirical tuning
    Two inconsistent formulas are given (Eq 10 and Sec 6.1); the schedule is not independently validated outside the paper's own experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 12988 in / 22965 out tokens · 258600 ms · 2026-08-04T17:11:31.757207+00:00 · methodology

0 comments
read the original abstract

We establish the first information-theoretic limits for multimodal retrieval. Casting ranking as lossy source coding, we derive a single-letter rate-distortion function $R(D)$ for reciprocal-rank distortion and prove a converse bound that splits into a modality-balanced term plus a skew penalty $\kappa\,\Delta H$ capturing entropy imbalance and cross-modal redundancy. We then construct an explicit entropy-weighted stochastic quantizer with an adaptive, per-modality temperature decoder; a Blahut-Arimoto argument shows this scheme achieves distortion within $O(n^{-1})$ of $R(D)$ using $n$ training triples. A VC-type analysis yields the first finite-sample excess-risk bound whose complexity scales sub-linearly in both the number of modalities and the entropy gap. Experiments on controlled Gaussian mixtures and Flickr30k confirm that our adaptive codes sit within two percentage points of the theoretical frontier, while fixed-temperature and naive CLIP baselines lag significantly. Taken together, our results give a principled answer to "how many bits per query are necessary" for high-quality multimodal retrieval and provide design guidance for entropy-aware contrastive objectives, continual-learning retrievers, and retrieval-augmented generators.

Figures

Figures reproduced from arXiv: 2509.11054 by Thomas Y. Chen.

Figure 1
Figure 1. Figure 1: Synthetic mixture: distortion (1−MRR) vs. rate. Curves are the mean of five runs; shaded bands show ±1 s.e [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 2 linked inside Pith

  1. [1]

    Emergence of invari- ance and disentanglement in deep representations.Journal of Machine Learning Research, 19(50):1–34, 2018

    Alessandro Achille and Stefano Soatto. Emergence of invari- ance and disentanglement in deep representations.Journal of Machine Learning Research, 19(50):1–34, 2018. 1, 7

    2018

  2. [2]

    The generalization ability of online algorithms for dependent data.IEEE Transactions on Information Theory, 59(1):573–587, 2012

    Alekh Agarwal and John C Duchi. The generalization ability of online algorithms for dependent data.IEEE Transactions on Information Theory, 59(1):573–587, 2012. 7

    2012

  3. [3]

    An algorithm for calculating the capacity of arbitrary discrete memoryless channels.IEEE Transactions on Information Theory, 18(1):14–20, 1972

    Shizuo Arimoto. An algorithm for calculating the capacity of arbitrary discrete memoryless channels.IEEE Transactions on Information Theory, 18(1):14–20, 1972. 4

    1972

  4. [4]

    Bartlett and Shahar Mendelson

    Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463–482, 2002. 5

    2002

  5. [5]

    Prentice-Hall, 1971

    Thomas Berger.Rate Distortion Theory: A Mathematical Basis for Data Compression. Prentice-Hall, 1971. 1, 2, 7

    1971

  6. [6]

    Richard E. Blahut. Computation of channel capacity and rate–distortion functions.IEEE Transactions on Information Theory, 18(4):460–473, 1972. 4

    1972

  7. [7]

    Concentration Inequalities: A Nonasymptotic Theory of In- dependence

    St ´ephane Boucheron, G ´abor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of In- dependence. Oxford University Press, 2013. 4

    2013

  8. [8]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InProceedings of the 37th Interna- tional Conference on Machine Learning, pages 1597–1607. PMLR, 2020. 1

    2020

  9. [9]

    Debiased contrastive learning

    Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, An- tonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. InAdvances in Neural Information Processing Sys- tems, 2020. 7

    2020

  10. [10]

    Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. Overview of the trec 2008 enterprise track.TREC, pages 1–13, 2008. 2

    2008

  11. [11]

    Multiterminal source coding under logarithmic loss.IEEE Transactions on Infor- mation Theory, 60(1):740–761, 2014

    Thomas Courtade and Sergio Verd ´u. Multiterminal source coding under logarithmic loss.IEEE Transactions on Infor- mation Theory, 60(1):740–761, 2014. 4

    2014

  12. [12]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas.Elements of Informa- tion Theory. John Wiley & Sons, 2nd edition, 2006. 1, 2, 3, 5, 7

    2006

  13. [13]

    Cambridge University Press, 2nd edition, 2011

    Imre Csisz ´ar and J ´anos K ¨orner.Information Theory: Cod- ing Theorems for Discrete Memoryless Systems. Cambridge University Press, 2nd edition, 2011. 2, 3

    2011

  14. [14]

    What to align in multimodal con- trastive learning? InInternational Conference on Learning Representations, 2025

    Benoit Dufumier, Javiera Castillo Navarro, Devis Tuia, and Jean-Philippe Thiran. What to align in multimodal con- trastive learning? InInternational Conference on Learning Representations, 2025. 7

    2025

  15. [15]

    Rate–distortion for ranking with incomplete information

    Farzad Farnoud, Moshe Schwartz, and Jehoshua Bruck. Rate–distortion for ranking with incomplete information. arXiv preprint arXiv:1401.3093, 2014. 7

    Pith/arXiv arXiv 2014

  16. [16]

    Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomas Mikolov

    Andrea Frome, Greg S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. InAdvances in Neural Information Processing Systems, pages 2121–2129, 2013. 1

    2013

  17. [17]

    Gray.Vector Quantization and Signal Compression

    Allen Gersho and Robert M. Gray.Vector Quantization and Signal Compression. Springer, 1992. 4

    1992

  18. [18]

    The information bottle- neck problem and its applications in machine learning.IEEE Journal on Selected Areas in Information Theory, 1(1):19– 38, 2020

    Ziv Goldfeld and Yury Polyanskiy. The information bottle- neck problem and its applications in machine learning.IEEE Journal on Selected Areas in Information Theory, 1(1):19– 38, 2020. 7

    2020

  19. [19]

    Gray and David L

    Robert M. Gray and David L. Neuhoff. Quantization in sig- nal processing.IEEE Transactions on Information Theory, 44(6):2325–2383, 1998. 4, 1

    1998

  20. [20]

    Dual alignment unsupervised domain adaptation for video–text retrieval

    Xiaoshuai Hao, Wanqian Zhang, Dayan Wu, Fei Zhu, and Bo Li. Dual alignment unsupervised domain adaptation for video–text retrieval. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18962–18972, 2023. 4

    2023

  21. [21]

    Towards the generalization of contrastive self- supervised learning

    Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the generalization of contrastive self- supervised learning. InProceedings of the 11th Inter- national Conference on Learning Representations (ICLR),

  22. [22]

    Cumulated gain- based evaluation of ir techniques.ACM Transactions on In- formation Systems, 20(4):422–446, 2002

    Kalervo J ¨arvelin and Jaana Kek ¨al¨ainen. Cumulated gain- based evaluation of ir techniques.ACM Transactions on In- formation Systems, 20(4):422–446, 2002. 2

    2002

  23. [23]

    Le, Yunhsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision.arXiv preprint arXiv:2102.05918, 2021. ICML 2021 version. 7

    Pith/arXiv arXiv 2021

  24. [24]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128–3137, 2015. 6

    2015

  25. [25]

    Generalization analysis for contrastive representation learning

    Yunwen Lei, Tianbao Yang, Yiming Ying, and Ding-Xuan Zhou. Generalization analysis for contrastive representation learning. InProceedings of the 40th International Confer- ence on Machine Learning. PMLR, 2023. 7

    2023

  26. [26]

    MIT Press, 2nd edition, 2018

    Mehryar Mohri, Afshin Rostamizadeh, and Ameet Tal- walkar.Foundations of Machine Learning. MIT Press, 2nd edition, 2018. 5

    2018

  27. [27]

    Rep- resentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Rep- resentation learning with contrastive predictive coding. In Advances in Neural Information Processing Systems, pages 10221–10234, 2018. 3

    2018

  28. [28]

    Estimation of entropy and mutual informa- tion.Neural Computation, 15(6):1191–1253, 2003

    Liam Paninski. Estimation of entropy and mutual informa- tion.Neural Computation, 15(6):1191–1253, 2003. 1

    2003

  29. [29]

    Quantization and the method ofk-means

    David Pollard. Quantization and the method ofk-means. IEEE Transactions on Information Theory, 28(2):199–205,

  30. [30]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Christopher Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR,...

    2021

  31. [31]

    A theoretical analy- sis of contrastive unsupervised representation learning

    Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analy- sis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference on Ma- chine Learning, pages 5628–5637. PMLR, 2019. 7

    2019

  32. [32]

    Cambridge University Press, 2014

    Shai Shalev-Shwartz and Shai Ben-David.Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014. 5

    2014

  33. [33]

    Claude E. Shannon. A mathematical theory of communica- tion. InBell System Technical Journal, pages 379–423. 1948. 2, 7

    1948

  34. [34]

    QUEST: Quadruple multimodal contrastive learning with constraints and self-penalization

    Qi Song, Tianxiang Gong, Shiqi Gao, Haoyi Zhou, and Jianxin Li. QUEST: Quadruple multimodal contrastive learning with constraints and self-penalization. InAdvances in Neural Information Processing Systems (NeurIPS), page to appear, 2024. 7

    2024

  35. [35]

    The information bottleneck method.37th Annual Allerton Con- ference on Communication, Control, and Computing, pages 368–377, 1999

    Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method.37th Annual Allerton Con- ference on Communication, Control, and Computing, pages 368–377, 1999. 7

    1999

  36. [36]

    Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere.Proceedings of the 37th International Con- ference on Machine Learning, 119:9929–9939, 2020. 3

    2020

  37. [37]

    Listwise approach to learning to rank: Theory and algorithm

    Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: Theory and algorithm. InProceedings of the 25th International Con- ference on Machine Learning (ICML), pages 1192–1199. ACM, 2008. 7

    2008

  38. [38]

    Explaining and mitigating the modality gap in contrastive multimodal learning.arXiv preprint arXiv:2412.07909, 2024

    Can Yaras, Siyi Chen, Peng Wang, and Qing Qu. Explaining and mitigating the modality gap in contrastive multimodal learning.arXiv preprint arXiv:2412.07909, 2024. 7

    arXiv 2024

  39. [39]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions.Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 6

    2014

  40. [40]

    Vector quantised contrastive learning

    Tianlong Yu, Xiaohan Xu, Phoebe Bromley, and Zachary Ding. Vector quantised contrastive learning. InAdvances in Neural Information Processing Systems, 2022. 7

    2022

  41. [41]

    On the gen- eralization of multi-modal contrastive learning

    Qi Zhang, Yifei Wang, and Yisen Wang. On the gen- eralization of multi-modal contrastive learning. InInter- national Conference on Machine Learning, pages 41677– 41693. PMLR, 2023. 7 Rate–Distortion Limits for Multimodal Retrieval: Theory, Optimal Codes, and Finite-Sample Guarantees Supplementary Material

    2023

  42. [43]

    The proof proceeds in four steps: S1

    Proof of Lemma 5.2: Finite–Sample Excess Distortion We supply the missing details behind theO(n −1)ex- cess–risk claim. The proof proceeds in four steps: S1. uniform concentration of the empirical entropies bHm; S2. stability of the bit–allocationR m and cell boundaries; S3. perturbation of the adaptive temperaturesτ ⋆ m; S4. Taylor expansion of the popul...

  43. [2023]

    Poster; openreview.net/forum?id=XDJwuEYHhme. 7