{"id":"642f111d-d222-4ce8-b6ff-3d640ad3333e","arxiv_id":"2505.22483","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors argue that modality collapse is caused by polysemantic neurons entangling noisy and predictive features across modalities, and that knowledge distillation or explicit basis reallocation frees rank bottlenecks to prevent it.","lead":"This paper proposes a mechanistic explanation for modality collapse in multimodal fusion, blaming noisy features from one modality entangling with predictive features from another in shared neurons. It also introduces an algorithm, Explicit Basis Reallocation (EBR), that improves robustness to missing modalities on MIMIC-IV and avMNIST.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Definition 1's conjugate features cannot exist for any label-informative y: nonnegative MI makes I(z;y)+I(z*;y)=0 force both terms to zero, so Theorem 1's premise is empty and the central collapse mechanism is unproven.","rationale":"The reader's weakest assumption is exactly right: Definition 1 is internally contradictory. This is the most load-bearing issue because Theorem 1 is the formalization of the abstract's central sentence; without a nonempty class of conjugate features, the claimed interference mechanism has no instances. The same vacuity also infects the proof, which silently adds geometric assumptions (opposite-direction activation, equal magnitudes) that are not implied by the definition. Theorems 2 and 3 have their own problems—for example, gamma(w)^(-1/n) and kappa^(-1/n) both converge to 1 as n grows, so the bounds are asymptotically vacuous, and Theorem 3 relies on an unproved 'weighted general case' of the Depth-Rank Duality result—but those flaws concern the rank-bottleneck story; the Definition 1 flaw attacks the causal mechanism itself. The empirical EBR results and the VIF/polysemanticity measurements in Appendix C are interesting and may stand independently, but they do not repair an inconsistent definition at the base of the theory. I therefore keep the reader's REJECT verdict: the theory, as written, does not support the explanation, though a revision that repairs Definition 1 and tightens the bounds could be worth reconsidering.","tokens_in":22840,"tokens_out":4238,"duration_ms":52595,"concrete_test":"Analytical re-derivation: take any feature z with I(z;y)>0 and any candidate z*; verify that I(z;y)+I(z*;y)=0 is unsatisfiable because both MI terms are nonnegative, so the set of conjugate features satisfying Definition 1 is empty whenever y is nontrivial. Then restate the intended mechanism as a pre-activation cancellation condition, w_p · z_y + w_p · z_epsilon = 0 with I(z_y;y)>0, and re-prove the limit in Theorem 1 from that condition alone. If the limit requires extra assumptions about superposition, orthogonality, or noise geometry that are absent from the paper, the central claim remains unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1, Definition 1 defines conjugate features by the pair of equalities I(z;y)+I(z*;y)=0 and I(zz*;y)=0, while requiring that at least one of z,z* has predictive value. Because mutual information is nonnegative, the first equality implies I(z;y)=I(z*;y)=0; combined with the second, the pair carries no label information either marginally or jointly. There is therefore no zy in Theorem 1 (a 'predictive conjugate feature') when the definition is taken literally. The proof of Theorem 1 then asserts, without derivation from Definition 1, that zy and its noisy conjugate zepsilon activate in opposite directions along the polysemantic subspace w_p with large equal magnitudes, so that their contributions cancel. That geometric cancellation is an additional assumption, not a consequence of the MI equalities. Since the paper's stated contribution is a theoretical mechanism for modality collapse, and the abstract's claim is directly formalized by Theorem 1, this empty premise is load-bearing: the interference mechanism is not established. A corrected definition (e.g., cancellation in the fusion pre-activation, w_p·z_y + w_p·z_epsilon ≈ 0, while allowing I(z_y;y)>0) might preserve the intuition, but as written the theorem has no subject matter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper aims to provide a mechanistic theory of modality collapse in multimodal fusion, arguing that noisy features from one modality become entangled with predictive features from another through shared polysemantic neurons in the fusion head, thereby suppressing the predictive contribution of the former modality. It formalizes this via conjugate features and an interference theorem (Theorem 1), links it to a low-rank simplicity bias through gradient-rank limits and a polysemantic bottleneck bound (Lemma 2, Theorem 2), and claims that cross-modal knowledge distillation frees rank bottlenecks (Theorem 3). On this basis it proposes Explicit Basis Reallocation (EBR), an adversarial encoder-level method, and reports improved missing-modality results on MIMIC-IV and avMNIST over several baselines.","tokens_in":23170,"tokens_out":6022,"duration_ms":70294,"significance":"If the theory were sound, the paper would offer a novel and practically relevant account of modality collapse, with a simple plug-in remedy (EBR) that is fusion-agnostic. The empirical portion is substantial: experiments span two datasets, multiple fusion backbones, missing-modality rates, noise perturbations, distillation-order ablations, and statistical significance tests; the rank and representation-similarity measurements are a useful contribution independently of the theory. However, the theoretical core is load-bearing for the paper's central claim, and the concerns raised by the stress-test analysis are confirmed by the manuscript text: Definition 1 is vacuous as written, the proof of Theorem 1 assumes the cancellation it purports to derive, and Theorems 2 and 3 contain bounds whose right-hand sides do not behave as claimed and whose key quantities are not well defined. The empirical results may stand on their own, but they cannot rescue the stated theoretical contribution.","major_comments":[{"comment":"The conjugate-feature definition is internally inconsistent and makes Theorem 1's premise empty. Since mutual information is nonnegative, the condition I(z;y)+I(z*;y)=0 forces I(z;y)=I(z*;y)=0, so neither z nor z* carries predictive information about y. This contradicts the requirement that at least one of z,z* has predictive value. The additional condition I(zz*;y)=0 is then redundant. Consequently, the object z_y ('predictive conjugate features') used in Theorem 1 does not exist under the stated definition, and the paper's central interference mechanism is not established. The definition would need to be reformulated (e.g., in terms of cancellation of pre-activations rather than mutual-information equalities) before Theorem 1 can have subject matter.","section":"Section 3.1 (Definition 1)"},{"comment":"The proof asserts, rather than derives, the geometric cancellation that drives the result. It states that because z_y and z_epsilon are conjugate, z_y activates in the opposite direction to z_epsilon along the polysemantic subspace w_p with large equal magnitudes, yielding w_p·z_y + w_p·z_epsilon = 0. Neither the sign opposition nor the equal-magnitude condition follows from Definition 1, which only imposes mutual-information equalities, nor from Lemma 3, which bounds a ratio of sums of dot products over weight subspaces. The limit in the theorem is therefore a restatement of the assumed cancellation, not a consequence of the preceding lemmas.","section":"Section 3.1 (Theorem 1 proof)"},{"comment":"The polysemantic bottleneck bound has several load-bearing defects. First, the left-hand side is not well defined: w is a subspace of the weight matrix, while the AGOP is a matrix, and no norm or embedding that places a subspace and a matrix in a common space is specified. Second, for any fixed gamma(w)>0, gamma(w)^{-1/n} tends to 1 as n grows, so the claimed bound becomes looser, not tighter, with training; it cannot establish that W converges to high-polysemanticity subspaces. Third, the chain K(1-2mu*lambda)^{nl} <= gamma(w)^{-1/n} in Eq. (4) compares a geometrically decaying quantity with a quantity that increases to 1, and no justification is given for the inequality. The interpretation in Section 4.2 that the rank drop 'is a consequence of the upper-bound in Theorem 2' is therefore not supported.","section":"Section 3.2 (Theorem 2)"},{"comment":"Theorem 3's conclusion is built into its construction rather than derived. In the proof, the authors assume |w∩X|→k_w and then define kappa=k_w/dim(w), which is exactly the degree of polysemanticity gamma(w) of the subspace. The claim that alignment shrinks polysemantic subspaces is thus an assumption about the boundedness of |w∩X|, not a consequence of knowledge-distillation dynamics. The theorem also inherits the vacuous-limit issue of Theorem 2, since kappa^{-1/n}->1 as n→∞. In addition, the 'weighted general case' of the Depth-Rank Duality is introduced in the proof without a formal statement or derivation, and the explicit form of alpha=h_phi(d(x~_i,x~_j)) and its monotonicity alpha1>alpha2>... are assumed without evidence.","section":"Section 3.3 (Theorem 3)"},{"comment":"Lemma 2 is not proved in the manuscript; its proof consists of two citations, one for rank decrease under SGD and one for convergence to the average gradient outer product. The statement rank(∇_l L_n) proportional to rank(Σ_x ∇φ_l(x)∇φ_l(x)^T) is not derived for the multimodal fusion architecture, and no rate or uniformity over n is given. Since Theorem 2 and the empirical rank-bottleneck interpretation in Section 4.2 depend on this lemma, the rank-bottleneck mechanism should be treated as conjectural rather than established.","section":"Section 3.2 (Lemma 2)"}],"minor_comments":[{"comment":"The phrase 'compete output basis' should read 'complete output basis'.","section":"Section 3.4"},{"comment":"The MUSE AUC-PRC entry is malformed ('0.39.87 ± 0.05' should presumably be '0.3987 ± 0.05').","section":"Table 1"},{"comment":"The notation I(zz*;y) is ambiguous; it should be written as I((z,z*);y) with the joint variable made explicit.","section":"Definition 1"},{"comment":"The quantity |w∩X| is not formally defined: w is a weight subspace and X is an input distribution, and no formal intersection operation is specified.","section":"Definition 2 and Theorem 2"},{"comment":"The limitation that Theorems 2 and 3 assume equal conditional cross-entropy reduction across features is acknowledged only in the conclusion; it should be stated as a hypothesis before the theorems are introduced.","section":"Section 5"},{"comment":"The lemma statement contains the typo 'predicitvity', and the proof invokes the Johnson–Lindenstrauss lemma through a reference to Elhage et al. (2022) without stating the specific property being used.","section":"Lemma 3"}],"recommendation":"reject","confidential_remarks":"The paper has a substantial and seemingly well-executed experimental component, and EBR may be a useful empirical contribution. However, the theoretical mechanism that the paper advertises as its main contribution is not sound in its current form: Definition 1 is vacuous, Theorem 1 assumes its conclusion, and the bounds in Theorems 2 and 3 do not support the stated convergence claims. These are not local presentation issues; they undermine the paper's advertised contribution. If the authors wish to resubmit, they would need to either rework the theory substantially or reframe the paper as an empirical study of basis reallocation for missing-modality robustness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The bottom line: this paper has a useful empirical story and a plausible algorithm, but its central theoretical claims, as written, do not survive contact with the definitions. I would not reject it outright—the experiments deserve a careful referee—but the theorems should not be the reason anyone trusts EBR.\n\nThe new thing here is the suggestion that modality collapse is caused by polysemantic neurons in the fusion head entangling noisy features of one modality with predictive features of another, and that KD or explicit basis reallocation (EBR) frees up rank bottlenecks to denoise. That is a genuinely different framing from earlier gradient-conflict or distributional hypotheses, and the EBR algorithm is concrete and agnostic to fusion strategy.\n\nThe experiments are the strongest part. On MIMIC-IV and avMNIST, the authors show rank decay around a critical β, improved noise robustness, and consistent gains in missing-modality settings across several fusion strategies. The statistical tests in Table 8 are a nice touch. If the mechanism works, it works regardless of the theory.\n\nThe soft spots are in the theory, and they are load-bearing. Definition 1 defines 'conjugate features' via I(z;y)+I(z*;y)=0 and I(zz*;y)=0, while requiring at least one of them to carry predictive value. Since mutual information is nonnegative, the first equality forces both terms to zero. So the predictive conjugate feature zy in Theorem 1 has no subject matter. The proof of Theorem 1 then simply asserts that zy and its noisy conjugate cancel along wp with 'large' opposite magnitudes—that is an extra assumption, not a consequence. Theorem 2's bound ||w - AGOP|| ≤ γ(w)^{-1/n} converges to 1 as n grows for any fixed γ, so it does not get tighter for polysemantic subspaces; the claimed implication in the text does not follow. Theorem 3 relies on a weighted extension of the Depth-Rank Duality result that is stated and not proved, and the constant κ is defined so that the bound becomes κ^{-1/n}—essentially the same quantity as the polysemanticity of w. Each of these is fixable, but as written they don't constitute a proof.\n\nWho is this for? Researchers working on missing-modality robustness will find the EBR results useful, and the polysemanticity-rank link is worth discussing. The theory needs repair before I would cite it.\n\nRecommendation: send it to peer review, but tell the reviewers to focus on whether the definitions and bounds can be corrected. If the authors fix Def. 1 (e.g., cancellation in the pre-activation while allowing I(z_y;y)>0) and prove rather than quote the bounds, the paper could be solid. As is, treat the theorems as conjectures and the experiments as the real contribution.","headline":"A useful empirical recipe (EBR) and a plausible mechanism, but the theoretical core has a vacuous definition and asymptotically vacuous bounds; treat the theorems as conjectures.","tokens_in":23641,"tokens_out":3985,"would_cite":false,"duration_ms":44207,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Modality collapse, the paper argues, happens when polysemantic fusion-head neurons entangle one modality's noisy features with another's predictive ones; freeing the resulting rank bottleneck—by distillation or explicit basis…","keywords":["modality collapse","multimodal fusion","polysemanticity","low-rank simplicity bias","knowledge distillation","missing modalities","basis reallocation"],"falsifier":"In a model that exhibits collapse, compute empirical mutual information between each fusion-head feature and the label: the theory requires a pair $(z, z^*)$ with $I(z;y)+I(z^*;y)=0$ while at least one is predictive, which non-negativity of mutual information makes impossible unless both are zero—so finding no such pair would empty Theorem 1 of its subject. Alternatively, train the identical fusion architecture with spectral or orthogonal regularization that provably removes the rank bottleneck: if collapse still occurs, rank starvation is not the driver the paper claims.","tokens_in":2258,"feed_emoji":"🧩","tokens_out":2926,"duration_ms":123267,"temperature":0.7,"pith_summary":"This paper tries to explain why multimodal fusion models often learn to rely on only a few of their input channels, a failure known as modality collapse. Its central claim is that collapse happens when noisy features from one modality become entangled with predictive features from another through shared neurons in the fusion head: the noise masks the predictive signal, so the entangled modality stops contributing to the loss and gets dropped. The paper further claims that this entanglement is a consequence of the low-rank simplicity bias of neural networks, which restricts gradient updates to a low-rank subspace and thereby forces fusion-head neurons to be polysemantic. It then shows that cross-modal knowledge distillation implicitly frees up these rank bottlenecks, and proposes an algorithm, Explicit Basis Reallocation (EBR), that does the same thing explicitly and can substitute for missing modalities at test time. If true, this turns a vaguely understood robustness failure into a mechanism that can be countered by a simple, fusion-agnostic change to the unimodal encoders.","feed_headline":"Noisy features explain why fusion models ignore modalities","feed_subtitle":"Modality collapse traces to polysemantic neurons and rank bottlenecks; reallocating basis dimensions reverses it.","key_machinery":"Polysemantic neurons—fusion-head neurons that encode features from more than one modality—are the mechanism's primary object. Definition 1 introduces 'conjugate features': a pair $z$, $z^*$ in the same modality, at least one predictive, whose mutual information with the label cancels when they are encoded together, i.e. $I(z;y)+I(z^*;y)=0$ and $I(zz^*;y)=0$; Theorem 1 claims these cancelling pairs are exactly what makes the entangled modality's marginal loss contribution vanish. Lemma 1 quantifies the collision rate: the fraction of cross-modal polysemantic neurons grows quadratically in the number of modalities. The rank bottleneck is the second load-bearing piece: Lemma 2 states that SGD gradient updates converge in rank toward the average gradient outer product (AGOP), and Theorem 2 bounds the distance between any weight subspace $w$ and the AGOP by $\\gamma(w)^{-1/n}$, tighter for more polysemantic subspaces—so SGD preferentially settles into polysemantic neurons. Theorem 3 says that once the encoders are aligned by knowledge distillation, that bound loosens to $\\kappa^{-1/n}$, polysemantic subspaces shrink, and monosemantic neurons become viable. Explicit Basis Reallocation carries the practical argument: an encoder-decoder pair per modality plus a modality discriminator, trained adversarially, so each modality's predictive features occupy the full output basis and noisy features get dedicated dimensions to be denoised along.","core_discovery":"On the paper's own terms: modality collapse is not primarily a symptom of conflicting gradients or data-distribution mismatch, but of an unintended cross-modal entanglement inside the fusion operator. Because neurons are polysemantic—each encoding features from more than one modality—and because the probability that a fusion-head neuron simultaneously encodes a noisy feature of one modality and a predictive feature of another grows quadratically with the number of modalities, the predictive features of the weaker modality get numerically cancelled by the noisy features entangled with them. Gradient updates under SGD are rank-constrained, converging toward the average gradient outer product (AGOP), which biases the fusion head toward exactly such low-rank polysemantic parameterizations. Cross-modal knowledge distillation implicitly loosens this bottleneck: as student representations are pulled toward the teacher's, the fusion head's polysemantic subspaces shrink, freeing dimensions along which the noisy features can be denoised independently. The paper's algorithm, Explicit Basis Reallocation (EBR), turns this implicit process into an explicit adversarial objective, and the resulting per-modality basis mapping supports test-time substitution of missing modalities.","pith_inferences":["Any intervention that raises the effective rank of gradient updates in the fusion head—wider heads, spectral or orthogonal regularization, gradient whitening—should also suppress collapse, and the phase-transition point observed near $\\beta=4$ in the paper's rank experiments should shift accordingly; this is a testable prediction the paper does not run.","The formal definition of conjugate features appears problematic: since mutual information is non-negative, $I(z;y)+I(z^*;y)=0$ forces both terms to zero, contradicting the requirement that at least one feature be predictive; the qualitative story of noisy interference may still be correct, but it needs a different formalization, for instance in terms of cancelled gradients rather than cancelled mu","The appendix's variance inflation factor measurements suggest a cheap monitoring proxy: if basis reallocation works by de-correlating features across modalities, then collapse severity should track measurable multicollinearity of the fused representation, and VIF or effective rank could serve as early-warning metrics during training.","The substitution intuition implies that the encoders share a latent factor space up to equivariance; a natural stress test the authors do not run is whether EBR's missing-modality ranking survives distribution shift between training and test missingness patterns."],"forward_implications":["Collapse should become more likely as the number of modalities grows, since cross-modal polysemantic collisions scale quadratically with the number of modalities; models with many input channels will need explicit rank management.","Distilling the surviving modality into the collapsed one, sequenced from weakest to strongest, should raise the effective rank of the student encoder and restore the collapsed modality's contribution; the paper reports this pattern on MIMIC-IV and avMNIST.","Because EBR modifies only the unimodal encoders, every fusion strategy should improve out-of-the-box when its backbones are swapped for KD or EBR versions; the paper reports such gains on MUSE, SMIL, ShaSpec, Grape, and M3Care.","The feature-to-basis mapping learned by EBR doubles as a substitutability ranking: when a modality is missing at test time, another modality's encoding can be projected through the decoder to stand in for it, which the paper validates on MIMIC-IV missingness rates up to 0.7."],"supporting_citations":[{"why":"Supplies the superposition mechanism of polysemantic neurons and the Johnson-Lindenstrauss argument that Lemmas 1 and 3 build on.","marker":"Elhage et al., 2022"},{"why":"Provides the neural capacity-allocation view of polysemanticity that the rank-bottleneck argument relies on.","marker":"Scherlis et al., 2022"},{"why":"Source of the low-rank simplicity bias claim that deep networks favour low-rank representations.","marker":"Huh et al., 2023"},{"why":"Establishes AGOP as the subspace network weights converge to under SGD; the target in Theorems 2 and 3.","marker":"Radhakrishnan et al., 2024"},{"why":"Shows SGD with weight decay minimizes weight-matrix rank, grounding Lemma 2's convergent rank.","marker":"Galanti et al., 2024"},{"why":"The Depth-Rank Duality result cited in the proofs of Theorems 2 and 3 to relate depth, rank, and empirical risk minimization.","marker":"Sreelatha et al., 2024"},{"why":"The prior conflicting-gradients explanation of modality collapse that this paper's mechanism complements and contrasts with.","marker":"Javaloy et al., 2022"},{"why":"Provides the MUSE baseline, the MIMIC-IV experimental setup, and the missing-modality evaluation protocol that EBR improves.","marker":"Wu et al., 2024"},{"why":"The contrastive distillation algorithm used to implement cross-modal knowledge distillation in the experiments.","marker":"Tian et al., 2020"},{"why":"Supplies the identifiability condition under which EBR's test-time modality substitution is justified.","marker":"Gulrajani & Hashimoto, 2022"}],"fun_headline_variants":["Polysemantic neurons betray weak modalities in fusion","Fusion head rank limits explain modality collapse","Explicit basis reallocation fixes modality dropout","Cross-modal distillation loosens fusion rank bottleneck","Modality collapse is polysemantic entanglement"],"cache_read_input_tokens":25728,"weakest_assumption_plain":"The central mechanism presupposes 'conjugate' feature pairs that are predictive yet cancel each other out in information-theoretic terms, and the rank-bottleneck bounds presuppose that every feature carries equal unique label information; if such pairs cannot exist or the equal-information condition fails, the collapse mechanism has no subject matter.","fun_headline_variants_meta":{"raw":{"variants":["Polysemantic neurons betray weak modalities in fusion","Fusion head rank limits explain modality collapse","Explicit basis reallocation fixes modality dropout","Cross-modal distillation loosens fusion rank bottleneck","Modality collapse is polysemantic entanglement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1592,"prompt_tokens":908,"completion_tokens":684,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":615}},"tokens_in":524,"tokens_out":684,"duration_ms":7222,"temperature":1.0,"reasoning_tokens":615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:06:26.558221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a model that exhibits collapse, compute empirical mutual information between each fusion-head feature and the label: the theory requires a pair $(z, z^*)$ with $I(z;y)+I(z^*;y)=0$ while at least one is predictive, which non-negativity of mutual information makes impossible unless both are zero—so finding no such pair would empty Theorem 1 of its subject. Alternatively, train the identical fusion architecture with spectral or orthogonal regularization that provably removes the rank bottleneck: if collapse still occurs, rank starvation is not the driver the paper claims.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the superposition mechanism of polysemantic neurons and the Johnson-Lindenstrauss argument that Lemmas 1 and 3 build on."},{"cited_title":"S., Gupte, A., and Poggio, T","cited_arxiv_id":null,"evidence_quote":"Shows SGD with weight decay minimizes weight-matrix rank, grounding Lemma 2's convergent rank."}],"review_version":1}