{"id":"16c0fc39-449a-436c-8561-e2810befb5e6","arxiv_id":"2505.24134","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A probabilistic tilting framework for contrastive learning yields closed-form Gaussian results showing which conditional statistics each loss can recover.","lead":"This paper recasts contrastive learning, the technique behind systems that align images with text, as building a joint probability distribution by tilting the product of single-modality distributions. It then derives exactly what standard and newly proposed contrastive losses can recover in the Gaussian case, and demonstrates the ideas on MNIST and an ocean-data task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Practical guidance rests on unnormalized linear encoders; standard CLIP's normalization may break the variance-inflation conclusions.","rationale":"The reader's weakest-assumption identification matches my own: the theorems apply to the exponential-tilting model class with linear, unnormalized encoders, while standard CLIP normalizes encoder outputs. My concern sharpens this into a concrete, testable gap between the theory and the practical message. I do not see an internal inconsistency in the Gaussian proofs that would overturn Corollaries 5.2, 5.4, or 5.7 within their stated scope; the algebraic typos noted in the proofs are real but do not change the minimizers. The paper is honest about the scope, and the normalized-encoder toy example is acknowledged as not general. Given this, the existing CONDITIONAL verdict remains appropriate: the central theoretical claims are sound for the stated model class, but the practical guidance should be qualified until the normalized-encoder case is studied beyond one dimension. No change to the reader's verdict is needed.","tokens_in":38838,"tokens_out":30185,"duration_ms":256964,"concrete_test":"Re-run the Section 5.4 two-dimensional Gaussian example with the standard CLIP objective (7) and normalized linear encoders g_u(u)=Gu/|Gu|, g_v(v)=Hv/|Hv|; estimate the conditional variances of the trained model by Monte Carlo and compare them with the true conditional variances and with Corollary 5.2's predicted variances (C_uu and C_vv). If the normalized model does not show strictly inflated conditional variances, the paper's practical guidance for CLIP fails in a simple non-toy case.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Corollaries 5.2, 5.4, and 5.7 are proved for the model classes in (39) and (46), where the encoders are linear and unnormalized. In actual CLIP implementations, encoders are L2-normalized as in (2), so the tilting is non-quadratic and the learned joint is non-Gaussian even for Gaussian data. The paper's only normalized-encoder analysis is the one-dimensional toy in Figure 6, which is not covered by any theorem and whose exact marginal matching is explicitly said not to be guaranteed in arbitrary dimensions. Consequently, the headline conclusion that the conditional loss inflates conditional variances while the joint loss better approximates marginals, and hence which loss to use for retrieval versus generative tasks, is not established for the standard normalized contrastive learning setting. This gap is load-bearing because the paper's abstract and discussion present the Gaussian results as informing practical algorithm choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a probabilistic reformulation of bimodal contrastive learning in which the learned representation is viewed as an exponential tilting of the product of the two marginal distributions, parameterized by encoders. It introduces generalized conditional and joint loss functions, as well as unnormalized and quadratic tiltings, and analyzes the resulting optimization problems in the multivariate Gaussian setting with linear encoders. The main theoretical results give closed-form minimizers: under the cosine-alignment conditional loss the conditional means are matched but conditional variances are inflated (Theorem 5.1, Corollary 5.2); under a one-sided positive-quadratic tilting the full conditional distribution for one modality is matched exactly (Theorem 5.3, Corollary 5.4); and under the joint loss the marginals are closer to the truth than under the conditional loss (Theorem 5.6, Corollary 5.7). The paper also includes numerical experiments on Gaussians, MNIST classification, and a Lagrangian data assimilation problem, and it connects the framework to retrieval and classification tasks.","tokens_in":39029,"tokens_out":19482,"duration_ms":166505,"significance":"If the results hold, the paper provides a useful mathematical lens on contrastive learning: the reformulation as a change of measure is elegant, and the closed-form Gaussian analysis gives concrete, falsifiable statements about which conditional statistics are matched by different losses and alignment metrics. The low-rank interpretation is a genuine contribution, and the numerical experiments, especially the data assimilation application, support the claimed relevance beyond a purely academic setting. The paper is not circular: the 'predictions' are theorems derived from the loss definitions, not fitted quantities. A notable strength is that the authors explicitly flag several limitations, including the restriction to unnormalized linear encoders and the lack of general guarantees for normalized encoders. The main weaknesses are algebraic errors in the appendix proofs and a scope mismatch between the unnormalized-encoder theory and the broader practical framing in the abstract and discussion.","major_comments":[{"comment":"The completion-of-the-square identity used to solve for A* is algebraically incorrect as printed. Expanding the displayed product Tr((A^T C_uu - C_vv^{-1}C_vu)(A C_vv - C_uu^{-1}C_uv)) gives Tr(A^T C_uu A C_vv) - 2Tr(A C_vu) plus a constant, whereas Lcond(A) is -Tr(A C_vu) + Tr(A^T C_uu A C_vv); in the scalar case the printed identity has cross term -2αc instead of -αc. The correct identity is Lcond(A) + (1/2)Tr(C_uu^{-1}C_uv C_vv^{-1}C_vu) = (1/2)||C_uu^{1/2}(A-A*)C_vv^{1/2}||_F^2 with A* = C_uu^{-1}C_uv C_vv^{-1}. The stated minimizer and the rank-r formula (44) are nevertheless correct, so this is a proof-repair issue rather than a wrong result.","section":"Appendix D, proof of Theorem 5.1, Eq. (68)"},{"comment":"The KL divergence formula has a sign error and a missing prefactor. The log-determinant term should be -log|(B+C_uu^{-1})C_{u|v}|, not +log|...|, and the quadratic term Δ(A,B) should carry the 1/2 factor from Lemma E.1. With the printed plus sign the objective is not bounded below in B, so the displayed derivation cannot identify B*; with the corrected sign, minimizing over S = B+C_uu^{-1} the function Tr(S C_{u|v}) - log|S C_{u|v}| gives S = C_{u|v}^{-1}, i.e. B* as in (49b). The missing 1/2 propagates into the rank-constrained objective (50b), which as printed does not define the minimizer of the actual loss. The central Corollary 5.4 uses the unconstrained minimizers and remains correct, but (50) needs to be rederived.","section":"Appendix D, proof of Theorem 5.3, Eq. (70)"},{"comment":"Corollaries 5.2, 5.4 and 5.7 are proved only for the unnormalized linear-encoder model classes (39) and (46). Section 5 states this restriction, and the normalized-encoder toy in Section 5.4 is explicitly described as not guaranteed to generalize to arbitrary dimensions. Nevertheless, the abstract, Section 1, and Section 7 present the Gaussian results as informing the choice of loss and alignment metric for contrastive learning generally, and the standard CLIP setting in (2) uses normalized encoders. Because normalization changes the learned measure qualitatively (the toy already shows non-Gaussian conditionals for Gaussian data), the paper should either extend the theory to normalized encoders or prominently restrict the practical recommendations to the unnormalized variants studied in the theorems.","section":"Scope of practical claims, Sections 5 and 7"}],"minor_comments":[{"comment":"The term 'Cvuv' is a typo and should read 'Cvu'.","section":"Corollary 5.2, Eq. (45b)"},{"comment":"The caption says 'cosine distance with the joint loss' but the described matching of the u|v conditional mean and variance is the positive quadratic form with the one-sided conditional loss of Theorem 5.3.","section":"Figure 3 caption"},{"comment":"The classification rule is written as an arg min of the log-weights; since wi is proportional to exp(⟨G_i, gu(u)⟩ + F_i), the mode should be an arg max.","section":"Equation (35)"},{"comment":"The line defining A* as C_vv^{-1}C_vu C_uu^{-1} is dimensionally the transpose of the minimizer stated in Theorem 5.1; the intended expression is C_uu^{-1}C_uv C_vv^{-1}.","section":"Appendix D, proof of Theorem 5.1"},{"comment":"The phrase 'supplementary meterial' should read 'supplementary material'.","section":"Section 1.3"},{"comment":"The labels 'Cosine Distance' are used for the unnormalized inner-product model (39); since cosine distance normally presupposes normalization, a brief clarification that these are unnormalized tiltings would help the reader map the theory to practice.","section":"Section 5 headings"}],"recommendation":"major_revision","confidential_remarks":"The algebraic errors in Appendix D are localized and the stated minimizers appear correct, so I do not think rejection is warranted. The larger risk is the gap between the unnormalized linear-encoder theory and the paper's framing around standard CLIP; this can be addressed by explicit qualification or by additional normalized-encoder analysis. The paper is a good fit for stat.ML and would make a useful contribution once the proofs are repaired and the scope is sharpened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi—\n\nThis paper gives a useful formalization of contrastive learning as a tilting of the product of marginals. The Gaussian analysis provides clean closed-form guidance: with cosine alignment and the two-sided conditional loss, you match conditional means but inflate variances; with a one-sided quadratic tilt, you match one full conditional exactly; with the joint loss, you approximate marginals better while biasing conditionals. The low-rank approximation perspective is a nice unifying view. The stated minimizers are correct, but the proofs in Appendix D contain real algebraic errors—a sign error in the log-determinant term in (70) and a dropped 1/2 factor in the quadratic term in the proof of Theorem 5.1. These are fixable and do not affect the results, but a referee should require corrected proofs.\n\nThe main caveat is the gap between the theory and standard CLIP. The Gaussian theorems apply to unnormalized linear encoders; CLIP uses L2-normalized encoders, which makes the tilting non-quadratic and the learned joint non-Gaussian. The paper is upfront in Section 5 that it works primarily with the unnormalized model and includes only a one-dimensional normalized toy example, with exact marginal matching not guaranteed in higher dimensions. Yet the abstract and discussion present the conclusions as informing practical algorithm choice. That is a stretch. The variance-inflation and marginal-approximation results should be labeled as properties of the stylized unnormalized model, not of standard CLIP. If the authors want to guide practice, they need either analysis of normalized models or a much clearer disclaimer.\n\nThe Lagrangian data assimilation experiment also overclaims: it is retrieval over a finite database with the true field coefficients as one embedding, not field reconstruction. It is a useful proof-of-concept for cross-modal retrieval, but it does not demonstrate data assimilation.\n\nOn the positive side, the framework is clean, the generalized tiltings and losses are well motivated, and the MNIST observation that standard classification is a one-sided conditional loss is a nice connection. The paper engages honestly with the literature and states its assumptions, even if it understates their restrictiveness. I would send it to a serious referee: it deserves careful review and will likely be cited for the closed-form minimizers and the conditional-versus-joint comparison. But the authors need to fix the proof errors and recalibrate the abstract and the DA claims.","headline":"A useful tilting framework with correct Gaussian minimizers, but the practical CLIP claims outrun the unnormalized-linear model, and the proofs need fixing.","tokens_in":39488,"tokens_out":8445,"would_cite":true,"duration_ms":72155,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard contrastive learning, read probabilistically, matches conditional means but inflates conditional variances; a one-sided quadratic tilt matches a full conditional and a joint loss improves marginals.","keywords":["contrastive learning","multimodal learning","conditional distributions","Gaussian analysis","low-rank matrix approximation","exponential tilting","crossmodal retrieval","probabilistic loss functions"],"falsifier":"Fit linear encoders to a two-dimensional centred Gaussian with $C_{uu}=C_{vv}=1$ and $C_{uv}=\\rho>0$ using the population two-sided cosine conditional loss, and inspect the learned conditional $\\nu_{u|v}$. The theory predicts variance exactly $1$ and mean $\\rho v$, so the variance exceeds the true conditional variance $1-\\rho^2$; observing a conditional variance smaller than $1$ under dependence would refute the central corollary.","tokens_in":38645,"feed_emoji":"📐","tokens_out":11443,"duration_ms":94156,"temperature":0.7,"pith_summary":"The paper's aim is to give contrastive learning a probabilistic meaning: training encoders is equivalent to picking an exponential tilting of the product of the two marginal distributions so that the resulting joint distribution approximates the true joint. In the Gaussian case with linear encoders this can be solved in closed form, turning latent-space identification into low-rank matrix approximation. The central finding is that the standard two-sided cosine loss matches conditional means but systematically inflates conditional variances, while a one-sided quadratic tilt can match one full conditional exactly and a joint loss brings the marginals closer to the truth. For a practitioner the upshot is that the choice of loss and alignment metric should depend on the task: retrieval wants correct conditional means, generation wants correct conditional spread.","feed_headline":"Contrastive learning matches conditional means, not variances","feed_subtitle":"A Gaussian analysis shows which loss to pick for retrieval versus generation.","key_machinery":"The machinery is the exponential tilting of the product of the marginals, $\\nu(du,dv)\\propto \\exp(\\langle Gu,Hv\\rangle)\\mu_u(du)\\mu_v(dv)$ for cosine alignment and $\\nu(du,dv)\\propto \\exp(-\\tfrac12|Gu-Hv|^2)\\mu_u(du)\\mu_v(dv)$ for the quadratic tilt, with $G,H$ linear encoders. These tiltings make the model joint Gaussian when the marginals are Gaussian, and all learnable statistics are carried by $A=G^\\top H$ and $B=G^\\top G$. The Gaussian analysis then reduces each learning problem to a low-rank matrix approximation in the Frobenius norm, solved by singular-value truncation; for the joint loss the per-singular-value function $h(\\sigma)$ encodes the optimal shrinkage of the cross-covariance.","core_discovery":"Within the Gaussian model class, the paper proves three precise statements. Under the cosine alignment with the two-sided conditional loss, the unique minimizer over unconstrained matrices is $A^*=C_{uu}^{-1}C_{uv}C_{vv}^{-1}$, and the induced conditionals $\\mathcal{N}(C_{uv}C_{vv}^{-1}v, C_{uu})$ and $\\mathcal{N}(C_{vu}C_{uu}^{-1}u, C_{vv})$ have the true conditional means but covariances strictly larger than the data conditionals unless $u$ and $v$ are independent. Under the one-sided quadratic tilt, minimizing the conditional KL divergence for $u|v$ alone yields parameters $(A^*,B^*)$ for which $\\nu_{u|v}$ is exactly $\\mathcal{N}(C_{uv}C_{vv}^{-1}v, C_{u|v})$. Under the cosine alignment with the joint KL loss, the optimal parameter applies the shrinkage function $h(\\sigma)=\\sigma^{-1}\\left(\\tfrac12\\sqrt{1+4\\sigma^2}-\\tfrac12\\right)$ to the singular values of the normalized cross-covariance, giving marginal covariances closer to the true marginals than those from the conditional loss. All three results have rank-constrained versions in which the best rank-$r$ encoder is obtained by singular-value truncation.","pith_inferences":["Editorial extension: Corollary 5.2 gives a diagnostic that can be run on any trained contrastive model: generate samples from the learned conditional and compare their spread with the spread of observed pairs; persistent over-dispersion would indicate the model is effectively solving the two-sided cosine conditional loss.","Editorial extension: the one-sided quadratic result suggests that in asymmetric modality pairs, such as text-to-image, the loss can be deliberately oriented to model the harder conditional exactly, accepting that the reverse conditional will be poorly calibrated; a symmetric two-sided quadratic objective appears not to be covered by the paper's closed-form theory.","Editorial extension: because the joint loss needs only one batch from the joint distribution and one from the product of marginals, it is computationally cheaper at scale; if the conditional-mean bias it introduces is tolerable, it may be a better default for generative fine-tuning, a conjecture that could be tested on non-Gaussian data."],"forward_implications":["For retrieval and mode-seeking, the standard two-sided cosine loss is well matched to the task: the optimal encoder recovers the true conditional means, so the mode of the learned conditional is the true conditional mode.","For generative sampling, the standard loss is systematically over-dispersed: the learned conditionals have the right centre but inflated spread, so samples are too scattered.","A one-sided quadratic tilt gives an exact conditional model for the single direction it targets, at the cost of leaving the other conditional misspecified; this suits applications with one dominant retrieval direction.","The joint loss is preferable when marginal structure matters, because it produces marginal covariances closer to the data, but it introduces bias in the conditional means.","The embedding dimension acts as a rank constraint: when $n_e<\\min(n_u,n_v)$, the optimal encoders are the best rank-$r$ approximations of the unconstrained solution, so capacity directly controls which conditional statistics are captured."],"supporting_citations":[{"why":"Von Neumann's trace inequality used in the proof of Theorem 5.6 to identify the optimal singular vectors of the joint-loss minimizer.","marker":"[3]"},{"why":"Supplies the generalized rank-constrained matrix approximation result used to write the rank-$r$ minimizers in Theorems 5.1, 5.3, and 5.6.","marker":"[12]"},{"why":"Frames Lagrangian data assimilation as a retrieval problem, providing the scientific application studied in Section 6.3.","marker":"[21]"},{"why":"Provides the MNIST classifier architecture whose cross-entropy objective Proposition 6.1 equates with a one-sided conditional loss.","marker":"[22]"},{"why":"Gives the Gaussian KL-divergence identity (Lemma E.1) used in the proof of Theorem 5.3.","marker":"[27]"},{"why":"Defines the standard image-text contrastive objective that the paper reformulates as the two-sided conditional loss in Theorem 2.8.","marker":"[28]"}],"fun_headline_variants":["Contrastive learning: conditional means, not variances","Gaussian view reveals which contrastive loss to pick","Retrieval versus generation: a contrastive loss guide","Optimal contrastive encoders match conditional means","Shrinkage rule fixes contrastive covariances"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The closed-form conclusions hold only inside the model class of exponential tiltings of the product of the marginals built from linear encoders; if the true conditionals are not representable by such a tilting, or if the encoders are nonlinear, the specific matched-statistics results in Corollaries 5.2, 5.4, and 5.7 do not apply.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive learning: conditional means, not variances","Gaussian view reveals which contrastive loss to pick","Retrieval versus generation: a contrastive loss guide","Optimal contrastive encoders match conditional means","Shrinkage rule fixes contrastive covariances"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1576,"prompt_tokens":1076,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":692,"tokens_out":500,"duration_ms":4641,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:34:40.722577+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit linear encoders to a two-dimensional centred Gaussian with $C_{uu}=C_{vv}=1$ and $C_{uv}=\\rho>0$ using the population two-sided cosine conditional loss, and inspect the learned conditional $\\nu_{u|v}$. The theory predicts variance exactly $1$ and mean $\\rho v$, so the variance exceeds the true conditional variance $1-\\rho^2$; observing a conditional variance smaller than $1$ under dependence would refute the central corollary.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Von Neumann's trace inequality used in the proof of Theorem 5.6 to identify the optimal singular vectors of the joint-loss minimizer."},{"cited_title":"and Torokhti, A","cited_arxiv_id":null,"evidence_quote":"Supplies the generalized rank-constrained matrix approximation result used to write the rank-$r$ minimizers in Theorems 5.1, 5.3, and 5.6."},{"cited_title":"and Jones, C","cited_arxiv_id":null,"evidence_quote":"Frames Lagrangian data assimilation as a retrieval problem, providing the scientific application studied in Section 6.3."},{"cited_title":"and Haffner, P","cited_arxiv_id":null,"evidence_quote":"Provides the MNIST classifier architecture whose cross-entropy objective Proposition 6.1 equates with a one-sided conditional loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Gaussian KL-divergence identity (Lemma E.1) used in the proof of Theorem 5.3."},{"cited_title":", Kim, J","cited_arxiv_id":null,"evidence_quote":"Defines the standard image-text contrastive objective that the paper reformulates as the two-sided conditional loss in Theorem 2.8."}],"review_version":1}