{"id":"04fb71e8-b214-46cc-a820-bc205b1dc20f","arxiv_id":"2501.11326","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Under conditional independence and marginal distribution assumptions, the inner product of unpaired modality representations is a monotone transform of the true likelihood ratio.","lead":"This paper proves that when AI models are trained on paired data like image-text and audio-text, their separately learned image and audio representations can be compared directly, even though image and audio were never paired in training. The authors also provide a Monte Carlo method that works when the direct comparison fails, and they test both on synthetic data and real vision, audio, and language models.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 2's stated g is not the derived expectation: the proof drops the uniform density's 1/Vol(S) factor and substitutes the dot product into a formula meant for ||phi_A+phi_C||; the stated g is not monotonic (e.g., d=2 gives an even function), so the Law as written is false.","rationale":"The paper's main theoretical object is Lemma 2. I checked the derivation against the vMF calculus and found two problems that are independent of whether Assumption 3 holds empirically. First, the expectation under U(S^{d-1}) carries a 1/Vol(S^{d-1}) prefactor; the proof drops it. This is not harmless: the stated g is meant to be a probability ratio, and without the prefactor and the K1*K2 constants from Assumption 2 the expression is not normalized. Second, the statement substitutes the inner product into a formula that is a function of kappa=||phi_A+phi_C||. The result is a g that is not monotonically increasing for even p and not even real for negative arguments when p is odd. The proof's final line also contains typos (phi_B instead of phi_C, missing factor 2). The intended corrected g(x)=1/C_p(sqrt(2+2x)) is monotone and probably what the authors meant, but it is not what Lemma 2 states. So the central 'Law' as written is false, and the strongest_claim as recorded cannot be accepted without revision. The reader's Assumption 3 concern remains valid: Fig. 2b and Fig. 7 show the dot-product setting violates the uniform-marginal assumption, so the practical scope of the Law is narrower than the title suggests. The Monte Carlo LogSumExp method (Lemma 1) does not rely on Assumption 3 and is supported by experiments (e.g., Fig. 3, Fig. 5), so the paper has independent value. I would keep the verdict conditional: require a corrected Lemma 2 with the proper normalization and g, plus a clear statement that the Law applies only when Assumption 3 holds.","tokens_in":16206,"tokens_out":20697,"duration_ms":202354,"concrete_test":"Analytically re-derive Lemma 2 for d=2 with normalized uniform p(phi_B). Compute E[exp(phi_A^T phi_B + phi_B^T phi_C)] as a function of t=phi_A^T phi_C. The paper's formula gives g(t)=2pi I_0(t); the correct formula is I_0(sqrt(2+2t))/I_0(1)^2 (up to the K constants from Assumption 2). Plot both on [-1,1]: the paper's g(t) is even and non-monotonic, while the corrected formula is strictly increasing. This single check settles whether Lemma 2's g is the actual probability ratio.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Lemma 2 as stated is not the function the proof derives. The proof evaluates E_{phi_B}[exp(phi_A^T phi_B + phi_B^T phi_C)] with p(phi_B)=U(S^{d-1}). The uniform density is 1/Vol(S^{d-1}), not 1. The vMF identity gives \\int_S exp(kappa mu^T x) dS = 1/C_p(kappa), hence the expectation is 1/(Vol(S) C_p(kappa)), not 1/C_p(kappa). The stated g(x)=(2pi)^{p/2} I_{p/2-1}(x)/||x||^{p/2-1} then substitutes the scalar inner product x into a formula meant for kappa=||phi_A+phi_C||. For p=2 this yields g(x)=2pi I_0(x), which is even and not monotonic; for odd p it is not real for negative x. The correct function depends on sqrt(2+2x) and is monotone. The equality also needs the omitted K1*K2 constants from Assumption 2 and the 1/Vol(S) prefactor; without them the right side is not a normalized probability ratio. Thus the central 'Law' in the exact form claimed is not established even when Assumptions 1-3 hold, independent of the separate empirical concern that Assumption 3 is violated for the dot-product critic (Fig. 2b, Fig. 7).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether contrastive representations learned from paired data (A,B) and (B,C) can support inferences about unpaired modalities A and C. Lemma 1 expresses the marginal likelihood ratio p(C|A)/p(C) as an expectation over representations of the bridge modality B under Assumptions 1 (conditional independence) and 2 (density-ratio critics). Lemma 2 aims to show that, under an additional uniformity assumption on the representation marginal, this ratio is a monotone function of the dot product phi_A(A)^T phi_C(C), thereby justifying the common 'direct comparison' heuristic. Lemma 3 gives an analogous closed form for unnormalized Gaussian representations with an L2 critic. Section 5 turns Lemma 1 into a Monte Carlo LogSumExp algorithm that avoids Assumption 3. Experiments on synthetic data, on CLIP/CLAP/LanguageBind, and in language-conditioned reinforcement learning compare direct comparison with the Monte Carlo method and with ground-truth oracles.","tokens_in":16504,"tokens_out":17499,"duration_ms":150219,"significance":"If Lemma 2 were correct, it would provide a theoretical grounding for a widely used heuristic. The Monte Carlo method based on Lemma 1 is a genuinely useful and assumption-light contribution: it connects pre-trained models without additional training and provides a principled treatment of ambiguity in language-conditioned tasks. The derivations are not circular: they start from Bayes' rule and assumptions adapted from prior work on InfoNCE and hyperspherical uniformity, and the experiments compare against ground-truth oracles. However, the central 'Law' is not established as stated. Lemma 2 contains a normalization error, an incorrect substitution into the Bessel function, and the printed function is not monotone. The paper's own experiments show that the key uniformity assumption is violated for the dot-product setting of Lemma 2. These issues affect the main theoretical claim and require substantial correction.","major_comments":[{"comment":"The claimed equality and the stated monotonicity are not correct as written. The proof writes the expectation under U(S^{d-1}) as an unnormalized surface integral; the uniform density contributes a factor 1/Vol(S^{d-1}), and the von Mises-Fisher normalization gives E[exp(phi_B^T(phi_A+phi_C))] = 1 / (Vol(S^{d-1}) C_p(||phi_A+phi_C||)), not 1/C_p(...). More importantly, the last step substitutes the inner product x = phi_A^T phi_C directly into the Bessel-function formula. The correct argument is kappa = ||phi_A+phi_C|| = sqrt(2+2x). For p=2 the printed g(x) = (2*pi)^(p/2) I_{p/2-1}(x)/||x||^{p/2-1} reduces to 2*pi*I_0(x), an even function, so it is not monotonically increasing on [-1,1]; for odd p it is not real for negative x. Thus the central 'Law' as stated is false even under Assumptions 1-3. The fix is to set g(x) = (K1*K2 / Vol(S^{d-1})) * (2*pi)^(p/2) I_{p/2-1}(sqrt(2+2x)) / (sqrt(2+2x))^{p/2-1}, to include the previously omitted constants, and to prove monotonicity of I_{p/2-1}(kappa)/kappa^{p/2-1} in kappa. The proof's final line also contains a typo: 'phi_A(a)^T phi_B(b)' should read 'phi_A(a)^T phi_C(c)'.","section":"4.3 (Lemma 2 and its proof)"},{"comment":"The value of delta is inconsistent between the main text and the appendix: Lemma 3 states delta = 1/(c+1), whereas Eq. (8) and the appendix's final line correspond to delta = 2/(c+1). The sentence after Lemma 3 that the log probability ratio 'asymptotically approaches the negative l2 distance' as c increases is also inaccurate: with gamma -> 1/4 and delta -> 0 as c -> infinity, the limiting log ratio is -1/4 ||phi_A - phi_C||^2 (up to an additive constant), not the negative l2 distance or the negative squared l2 distance used by the critic. Please correct the constant in the lemma statement and revise the asymptotic claim.","section":"4.4 / Appendix B (Lemma 3)"},{"comment":"The paper's own experiments show that Assumption 3 is violated for the dot-product critic, which is precisely the setting of Lemma 2. Figure 2b shows that the Direct method underperforms the Monte Carlo method, and Fig. 7 shows that the unnormalized representation distribution is not uniform. The paper therefore does not empirically validate the 'Law' in its main claimed form; it validates only the conditional statement 'if Assumption 3 holds, then the Law holds.' Moreover, Section 6.2.2 tests uniformity only for language representations over the AudioSet ontology, not for the marginal distributions of all three encoders used in the dot-product experiments. Please either provide experiments in which Assumption 3 is verified for the dot-product critic, or explicitly restrict the claim of the 'Law' to settings where uniformity can be established, presenting the synthetic dot-product results as a negative test of the assumption.","section":"6.1.1 (Fig. 2b, Fig. 7)"}],"minor_comments":[{"comment":"The displayed lower bound log E[exp(phi_A^T phi_B + phi_B^T phi_C)] >= phi_A^T phi_C is not a consequence of the triangle inequality and is false for a pointwise realization. For example, in d=2 take phi_A=(1,0), phi_C=(0,1), and phi_B=(-1,-1)/sqrt(2); then phi_B^T(phi_A+phi_C) = -sqrt(2) < 0 = phi_A^T phi_C. The passage should be corrected or removed.","section":"4.2"},{"comment":"Assumption 3 is stated with an integral over an indicator p(phi) = integral p(x) 1{phi_M(x)=phi} dx, which is not well-defined for continuous representations; the assumption should be stated as a pushforward measure on the hypersphere.","section":"3.3 (Assumption 3)"},{"comment":"The two-sample Kolmogorov-Smirnov results (CLIP p=0.0877, CLAP p=0.1788) are described as showing that Assumption 3 'fares well,' but a failure to reject uniformity is not positive evidence for uniformity, especially with a small ontology. Reporting an effect size or a power analysis would make the claim more appropriate.","section":"6.2.2"},{"comment":"The caption lists 'Center' twice; the second center should be 'Right'.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The main theoretical claim is not usable in its current form, but the error in Lemma 2 appears fixable by a corrected definition of g and a proper monotonicity proof. The Monte Carlo method (Lemma 1 + Section 5) is the strongest part of the paper and should be foregrounded. I recommend major revision rather than rejection, provided the authors correct Lemma 2, align Lemma 3 with its proof, and substantially soften the empirical claims about the 'Law' for the dot-product critic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely useful practical contribution and a clean first lemma, but the headline theoretical result is not as stated. Lemma 1 is correct: under conditional independence and the density-ratio assumption, the marginal ratio is an expectation over the bridge modality, and the Monte Carlo (LogSumExp) estimator is an unbiased approximation with no circularity. The experiments on CLIP/CLAP and the RL tasks show the method works in realistic settings, and the paper is honest about testing its assumptions.\n\nThe problem is Lemma 2. The proof writes the expectation under the uniform distribution on the sphere but drops the 1/Vol(S) factor, then substitutes the scalar inner product x directly into the vMF normalizing constant formula. That substitution is wrong: κ is the norm of the sum of two unit vectors, √(2+2x), not x itself. As stated, g(x) for d=2 is proportional to I_0(x), which is even and not monotone, and for odd d it's not real for negative x. So the 'Law' as written is false, even granting Assumptions 1–3. The intended corrected function—g depending on √(2+2x)—would be monotone, so the mistake looks fixable, but the paper currently asserts an unproven and incorrect claim.\n\nThere's also a smaller inconsistency in Lemma 3: the lemma states δ = 1/(c+1), but the appendix proof derives δ = 2/(c+1). A referee should ask for a reconciliation.\n\nEmpirically, the paper's own Fig 2b shows the direct comparison fails for the dot-product critic exactly because Assumption 3 is violated; Fig 7 confirms the non-uniformity. So the 'Law' does not hold in the very setting it is meant to justify. The Monte Carlo method avoids Assumption 3 and performs well, and that is the part worth taking seriously.\n\nWho gets value? Researchers interested in combining pre-trained contrastive models or handling ambiguity in RL. They should use the LSE method and cite the paper for that, not for the 'Law'. The paper deserves a serious referee: the core idea is important, the errors are fixable, and the experimental evidence for the practical method is solid. I would send it to review with a request to fix Lemma 2 and the Lemma 3 constants, and to soften the claims about the 'Law' given their own empirical violations.\n\nFor a reading group, maybe—the flawed theorem makes for a good discussion on vMF integrals. I'd cite the LSE method in future work.","headline":"The practical Monte Carlo method is solid, but the paper's central 'Law' as stated is not actually proven—the stated g(x) is not the function the derivation yields, and the paper's own experiments show the needed uniform-marginal assumption fails for the dot-product critic.","tokens_in":17000,"tokens_out":4919,"would_cite":true,"duration_ms":44961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that, under three stated assumptions, the probability ratio between two modalities never seen together in training is a deterministic monotone function of the dot product of their contrastive representations.","keywords":["contrastive learning","multimodal alignment","density ratio estimation","InfoNCE","representation geometry","Bayesian marginalization","language-conditioned reinforcement learning","zero-shot retrieval"],"falsifier":"Estimate $p(C\\mid A)/p(C)$ directly from data, for example by counting matched examples in a large synthetic triple satisfying conditional independence, then compare it with $g(\\phi_A(A)^\\top \\phi_C(C))$ for the same inputs. If the two disagree beyond sampling error, or if the empirical relation is not monotone in the dot product, the law as stated fails. The paper's Figure 2b already points toward this test: under the dot-product critic the direct method underperforms the Monte Carlo baseline, which is evidence that the uniformity assumption fails in that setting.","tokens_in":15994,"feed_emoji":"🔗","tokens_out":7693,"duration_ms":71421,"temperature":0.7,"pith_summary":"Contrastive models are often used as if a representation from one modality can stand in for another, even when the two modalities never appeared together in training. This paper asks when that 'plug-and-play' swap is actually doing correct probabilistic inference. Under three assumptions, it proves the probability ratio $p(C\\mid A)/p(C)$ between unpaired modalities is a deterministic, monotonically increasing function of the dot product $\\phi_A(A)^\\top \\phi_C(C)$, justifying the common heuristic. It also gives a Monte Carlo estimator that marginalizes over the intermediate modality under fewer assumptions, and shows this estimator composes pretrained models and improves language-conditioned navigation under ambiguous instructions.","feed_headline":"Unpaired modality embeddings are likelihood ratios under a new law","feed_subtitle":"A proof of when plug-and-play modality swapping works, plus a Monte Carlo method for when it fails.","key_machinery":"The machinery is a chain of three objects. First, Assumption 2 treats each contrastive critic as a density-ratio estimator, $e^{f(\\phi_A,\\phi_B)} \\propto p(B|A)/p(B)$. Second, Lemma 1 rewrites the target ratio as an expectation over the intermediate representation, $p(C|A)/p(C) = K_1 K_2 \\, \\mathbb{E}_{\\phi_B}[\\exp\\{f(\\phi_A,\\phi_B)+f(\\phi_B,\\phi_C)\\}]$, which is exact message passing on the graphical model $A-B-C$ when Assumption 1 holds. Third, under Assumption 3 the uniform marginal turns that expectation into an integral of a von Mises-Fisher density, whose normalizing constant yields the monotone Bessel function $g$. When uniformity fails, the same expectation can be approximated by Monte Carlo samples, giving the LogSumExp algorithm.","core_discovery":"The central result is Lemma 2: when A and C are conditionally independent given B, when the two contrastive critics encode density ratios, and when all representation marginals are uniform on the unit hypersphere, the ratio $p(C\\mid A)/p(C)$ equals $g(\\phi_A(A)^\\top \\phi_C(C))$, where $g(x) = (2\\pi)^{p/2} I_{p/2-1}(x)/\\|x\\|^{p/2-1}$ is monotonically increasing in $x$. In words, a direct dot-product comparison of embeddings from modalities never trained together is not just a similarity score; it is a fixed monotone transform of the true likelihood ratio. The paper also proves a Gaussian analogue for unnormalized representations under an L2-distance critic, and derives a Monte Carlo LogSumExp estimator from Lemma 1 that replaces the uniformity assumption with explicit sampling over the intermediate modality.","pith_inferences":["A practical diagnostic suggested by the two estimators: run both direct comparison and the LogSumExp estimator on a held-out set; where their rankings diverge, the uniformity assumption is suspect and the Monte Carlo answer is safer, since it requires fewer assumptions.","The framework points toward graph-structured modality composition beyond a single chain: chaining through several intermediate modalities should work whenever each step satisfies conditional independence, though violations would accumulate, and the Monte Carlo estimator degrades gracefully rather than failing outright.","The language-conditioned RL results suggest that any policy using a single averaged language embedding for an ambiguous instruction is discarding a multimodal goal distribution; replacing the mean with an explicit sum over candidate goal states is a drop-in improvement.","Because the law's closed form is monotone, the paper's result also explains why uncalibrated cosine-similarity thresholds in existing systems can still produce correct rankings: the threshold scale is wrong, but the ordering is preserved."],"forward_implications":["When the assumptions hold, cosine similarity between unpaired modality embeddings is a monotone transform of the log-likelihood ratio, so retrieval by nearest-neighbor search is implicitly Bayesian.","The Monte Carlo LogSumExp method gives a way to compose pretrained contrastive models across a shared intermediate modality without needing paired data from the target modalities.","For language-conditioned reinforcement learning, marginalizing over future states rather than using a mean language embedding avoids the failure mode of navigating to the centroid of an ambiguous description.","With an L2-distance critic and Gaussian marginals, the same reasoning yields a closed form in which the log ratio is essentially the negative squared distance between unpaired representations.","Because $g$ is monotone, ranking candidates by raw dot product agrees with ranking by the exact probability ratio, so the method validates common practice even when users never compute the Bessel transform explicitly."],"supporting_citations":[{"why":"Supplies the density-ratio interpretation of contrastive learning used as Assumption 2.","marker":"(Poole et al., 2019)"},{"why":"Supplies the uniformity-on-the-hypersphere assumption and its theoretical motivation for Assumption 3.","marker":"(Wang & Isola, 2020)"},{"why":"Provides consistency results for noise contrastive estimation that underpin Assumption 2.","marker":"(Ma & Collins, 2018)"},{"why":"Provides CLIP, the pretrained vision-language model used in the real-world audio-visual experiments.","marker":"(Radford et al., 2021)"},{"why":"Provides CLAP, the pretrained audio-language model used to bridge audio and image through language.","marker":"(Elizalde et al., 2023)"},{"why":"Provides LanguageBind, a multi-modal model whose direct comparison performance is used to validate the law and as a baseline.","marker":"(Zhu et al., 2023)"},{"why":"Provides ImageBind, used in experiments that swap the intermediate modality to test the framework's generality.","marker":"(Girdhar et al., 2023)"},{"why":"Supplies the Gaussian marginal assumption used in the unnormalized-representation extension.","marker":"(Eysenbach et al., 2024)"},{"why":"Supplies the contrastive reinforcement learning formulation used in the language-conditioned navigation experiments.","marker":"(Eysenbach et al., 2023)"},{"why":"Provides the AudioSet ontology and dataset used in the real-world evaluation of audio-visual retrieval.","marker":"(Gemmeke et al., 2017)"}],"fun_headline_variants":["New law: unpaired embeddings recover likelihood ratios","Dot products of unpaired embeddings proven to be likelihood ratios","When unpaired modalities align: a probabilistic law","Contrastive law: unpaired dot products equal likelihood function","Proof: unpaired embeddings match likelihood ratios under conditions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that contrastive representations are spread uniformly over the unit sphere; the paper's own experiments show this is violated for the dot-product critic in which the closed-form law is stated.","fun_headline_variants_meta":{"raw":{"variants":["New law: unpaired embeddings recover likelihood ratios","Dot products of unpaired embeddings proven to be likelihood ratios","When unpaired modalities align: a probabilistic law","Contrastive law: unpaired dot products equal likelihood function","Proof: unpaired embeddings match likelihood ratios under conditions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2057,"prompt_tokens":907,"completion_tokens":1150,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1073}},"tokens_in":523,"tokens_out":1150,"duration_ms":10671,"temperature":1.0,"reasoning_tokens":1073,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:23:46.551494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Estimate $p(C\\mid A)/p(C)$ directly from data, for example by counting matched examples in a large synthetic triple satisfying conditional independence, then compare it with $g(\\phi_A(A)^\\top \\phi_C(C))$ for the same inputs. If the two disagree beyond sampling error, or if the empirical relation is not monotone in the dot product, the law as stated fails. The paper's Figure 2b already points toward this test: under the dot-product critic the direct method underperforms the Monte Carlo baseline, which is evidence that the uniformity assumption fails in that setting.","supporting_citations":[{"cited_title":"Contrastive learning as goal-conditioned reinforcement learning, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive reinforcement learning formulation used in the language-conditioned navigation experiments."}],"review_version":1}