{"id":"57d6af68-d1d3-4259-8882-438b9acf1d65","arxiv_id":"2506.11465","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RollingQ rotates the classification query in a multimodal Transformer toward a rebalanced direction so attention stops over-favoring a single modality, restoring dynamic fusion and improving accuracy.","lead":"This paper shows that attention-based fusion in multimodal Transformers tends to lock onto one modality, ignoring the other even when it becomes uninformative. It proposes RollingQ, a simple query-rotation method that rebalances attention and improves accuracy on several audio-visual and vision-language benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's derivation equates softmax attention mass with the sum of raw logits (Eq. 4), so the AIR indicator and rebalance anchor may be miscalibrated; this needs an empirical proxy check.","rationale":"The central claim is that a self-reinforcing cycle creates a key-distribution gap and that rotating the query toward a rebalance anchor breaks the cycle. The paper offers strong empirical evidence for the phenomenon (noise sensitivity, gradient evolution, restored dynamics) and a reproducible codebase, so the method itself is likely to provide gains. The most load-bearing weakness is the theoretical bridge between average-key cosine similarity and true softmax attention allocation. The reader identified the norm-equality and average-key-faithfulness assumptions; our reading goes one step further and notes that even with equal norms and lengths, the sum of raw logits in Eq. 4 is not the softmax attention mass. This makes the derivation in Section 3.1-3.3 internally imprecise, not merely dependent on an empirical regularity. However, the empirical correlation observed by the authors could still be high in their specific models, so the concern is testable rather than a demonstrated failure. A proxy-validation check and a random-rotation control would settle whether the proposed anchor is causally responsible for the reported improvements. Given the paper's conditional acceptance already hinges on the soundness of its mechanism, our finding does not move the verdict; it sharpens the condition under which the paper should be accepted.","tokens_in":17988,"tokens_out":10454,"duration_ms":104410,"concrete_test":"On a trained Vanilla MT (e.g., Kinetic-Sound), compute per-sample true softmax attention mass A_m = \\Sigma_{j\\in m} \\exp(q\\cdot k_j/\\sqrt{d}) / \\Sigma_{\\text{all }l} \\exp(q\\cdot k_l/\\sqrt{d}) for both modalities. Compute the paper's proxy P_m = (L_m/\\sqrt{d}) ||q|| ||\\hat{k}_m|| \\cos\\theta_m (or simply the cosine similarity between q and the average key), following Eq. 4. Then measure the Spearman rank correlation between P_a - P_v and A_a - A_v over the validation set, and report the fraction of samples where sign(AIR) matches sign(A_a - A_v). If the correlation is weak or the sign agreement is low (e.g., <80%), the AIR indicator and the anchor q_b are not faithful summaries of attention allocation, and the claimed mechanism for RollingQ is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that \"attention score for modality m is determined by \\Sigma_j q k^m_j / \\sqrt{d}\". This is not correct: the post-softmax attention mass allocated to modality m is \\Sigma_{j\\in m} \\exp(q\\cdot k_j/\\sqrt{d}) / \\Sigma_{\\text{all }l} \\exp(q\\cdot k_l/\\sqrt{d}). Equation 4 rewrites the sum of logits, not the softmax allocation. The self-reinforcing-cycle analysis (Section 3.2) and the AIR indicator (Section 3.3, Eq. 9) therefore rest on the assumption that average-key cosine similarity tracks true softmax allocation. That assumption can fail even when ||k^a||\\approx||k^v|| and L_a\\approx L_v: a modality whose keys have a few very large logits can dominate softmax while having a comparable sum of logits to a modality with many moderate logits. Because the rebalance anchor q_b in Eq. 10 is built from normalized average keys, and the rotation target is chosen to reduce AIR, a miscalibrated proxy would not guarantee that the rotated query favors the unbiased modality. The empirical noise tests in Section 4.3 show a correlation, but they do not establish that the proxy is faithful across the sample distribution. This is a load-bearing correctness risk for the paper's diagnosis and for the transferability of RollingQ.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies why the attention mechanism in multimodal Transformers loses its dynamic adaptability, leading the model to favor one modality regardless of input quality. The authors argue that this bias is caused by a self-reinforcing cycle: the favored modality receives more attention, which amplifies its encoder gradients, which in turn further improves that modality's features and widens the gap between the attention-key distributions of different modalities. They propose Rolling Query (RollingQ), which detects imbalance with an Attention Imbalance Rate (AIR) indicator based on cosine similarity between the query and average keys, and then rotates the query toward a rebalance anchor that gives more weight to the under-attended modality. Experiments on CREMA-D, Kinetic-Sound, and CMU-MOSEI, plus noise-perturbation, correlation, and OOD tests, are presented as evidence that RollingQ restores dynamic attention and improves accuracy over vanilla and several specialized baselines.","tokens_in":18309,"tokens_out":3472,"duration_ms":39146,"significance":"If the central claim holds, the paper identifies a plausible and practically important failure mode of attention-based multimodal fusion and offers a remarkably simple intervention. The strength of the paper is that the proposed method is lightweight, the empirical demonstrations include a useful battery of diagnostic tests (noise corruption, Pearson correlation of attention with input quality, QUAG-style ablations, and OOD benchmarks), and the source code is promised. The claim that attention in multimodal Transformers becomes locked by a training feedback loop, and that a single query rotation can unlock it, is interesting and falsifiable. However, the paper's theoretical derivation is informal, the main accuracy numbers are reported without variance or multiple seeds, and the AIR indicator rests on a proxy that is not rigorously validated. The strongest baseline, MBT, still outperforms RollingQ on Kinetic-Sound and ties it on the other two datasets, so the practical edge over sophisticated baselines is not established even if the mechanism is correct.","major_comments":[{"comment":"The derivation equates the attention score for a modality with the sum of raw logits. This is not the post-softmax attention mass: the actual mass allocated to modality m is sum_{j in m} exp(q.k_j/sqrt(d)) divided by the sum over all tokens. Equation (4) rewrites the sum of raw logits, not the softmax allocation. Consequently, the AIR indicator in Eq. (9), the rebalance anchor in Eq. (10), and the rotation target all depend on the assumption that average-key cosine similarity tracks true softmax allocation. Even if ||k^a|| approx ||k^v|| and L_a approx L_v, this can fail: one modality with a few very large logits can dominate softmax while having a comparable sum of raw logits to a modality with many moderate logits. The noise-correlation experiments in Section 4.3 are suggestive but do not establish that the proxy remains faithful across the sample distribution or across different architectures. This is a load-bearing correctness risk; the paper should either reformulate AIR directly in terms of actual attention weights, or empirically validate the proxy, for example by reporting the correlation between the average-key cosine and the true softmax attention mass per sample.","section":"Table 1 and Section 4.2"},{"comment":"The reported accuracy improvements are not accompanied by standard deviations, confidence intervals, or multiple seeds. Some of the claimed gains are very small, e.g., +0.1 on MulT+RollingQ and +0.5 on Vanilla MT+RollingQ for CMU-MOSEI, and it is possible these are within run-to-run noise. The hyperparameters beta, rho, and the maximum rotation count are dataset-dependent, and the maximum rotation count is explicitly set to 1 or 3 per dataset in Appendix B. Without a sensitivity analysis or repeated-run statistics, the 'consistent accuracy gains' claim is not yet supported at the level the paper asserts. At minimum, the authors should provide mean and standard deviation over three or more seeds for the main comparison table and a sensitivity analysis for beta, rho, and rotation count.","section":"Table 1 and Section 4.2"},{"comment":"The self-reinforcing-cycle argument is presented as a theoretical analysis, but Eq. (8) does not rigorously imply that the modality receiving higher attention always obtains larger encoder gradients. The derivative of the softmax term through the attention weights can be negative for some tokens, and the gradient for modality m also includes the value-path term whose magnitude depends on the full attention distribution, not only on the total attention mass of that modality. The visualizations of gradient norms in Appendix A.2 are consistent with the proposed mechanism, but they are also consistent with other explanations such as differing learning dynamics of the two encoders. The paper should either formalize the cycle more carefully or explicitly label it as an intuitive mechanism supported only by empirical monitoring.","section":"Section 3.2, Eq. (8)"}],"minor_comments":[{"comment":"There are several typos and awkward phrasings: 'aims to to address', 'we has E[Q]=0', 'the the gradient', 'statisfied', and an inconsistent reference to the code repository as 'Github' without a URL in the abstract. These should be cleaned up.","section":"Abstract and Introduction"},{"comment":"Equation (4) includes the sequence length L_m and the average key, but the text does not explain why the softmax denominator is ignored in the definition of 'attention score for modality m'. A sentence clarifying that this is a proxy, or a redefinition, would improve readability and prevent the misconception that it gives exact softmax mass.","section":"Section 3.1"},{"comment":"The multi-layer extension trains the first attention block while 'temporarily dropping the second block' for several epochs, then switches to the second block. This is an unusual training schedule and the description is too brief to be reproducible; more detail on the number of epochs, how the dropped block is reintroduced, and how the AIR indicator is computed for the second block after switching is needed.","section":"Appendix B"},{"comment":"The complexity analysis reports parameters and GFLOPs for CREMA-D, but the '1% increase in parameters' appears inconsistent with the table, which shows 59.87M to 60.46M, an increase of about 1% that checks out, while the GFLOPs increase is 0.07, about 0.005%. The text should state these numbers precisely rather than rounding to 0.1%, and the source of the GFLOPs value should be clarified.","section":"Table 4"},{"comment":"The paper claims RollingQ achieves 'comparable results' to static fusion imbalance techniques and 'strong performance' relative to MBT and JMT, but MBT outperforms RollingQ on Kinetic-Sound and ties on CREMA-D and MOSEI. The discussion should acknowledge this directly rather than implying superiority.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is presented as an ICML 2025 proceedings paper; I am reviewing it as a submitted manuscript. The strongest contribution is the diagnostic observation and the very simple intervention, but the theoretical framing is currently too loose and the empirical evidence lacks statistical grounding. The central claim is defensible, and the requested revisions are concrete: validate or replace the AIR proxy, add error bars and sensitivity analysis, and tighten the gradient argument. If those are addressed, I would be willing to reconsider. I do not see grounds for rejection, because the phenomenon and intervention are both interesting and potentially useful even if the formal derivation remains imperfect."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper that deserves a serious referee, not a desk reject. It identifies a real failure mode—attention in multimodal Transformers locks onto one modality regardless of input quality—and proposes a cheap query-rotation fix that helps consistently, if modestly, across several datasets. Code is available, and the empirical work is genuinely broad: noise robustness, OOD, ablations, complexity numbers, and an honest limitations section.\n\nWhat's actually new: the diagnosis that average attention keys diverge across modalities in a self-reinforcing loop, and the RollingQ rotation mechanism used to break that loop. That is a legitimate extension of prior imbalance work like OGM and PMR, which mostly targeted static fusion; this paper moves the discussion into dynamic attention and adds a simple intervention.\n\nThe main soft spot is the theory, and it is not just cosmetic. Section 3.1 states that the attention score for a modality is determined by the sum of raw logits (Eq. 4), but softmax attention mass is the exponentiated logit sum normalized over all tokens. The AIR indicator and the rebalance anchor are built on average-key cosine similarity, which tracks the logit sum, not the softmax allocation. The two can disagree even when key norms and sequence lengths are equal—a few very large logits in one modality can dominate softmax while its total logit sum is comparable to a modality with many moderate logits. The empirical noise tests show correlation, but they don't prove the proxy is faithful across the distribution. So the diagnosis is plausible but not rigorously established.\n\nOther issues are in proportion. No error bars anywhere. Hyperparameters β, ρ, and the max rotation count are tuned per dataset (1 for CREMA-D/MOSEI, 3 for Kinetic-Sound). The strongest baseline, MBT, beats RollingQ on some datasets, so the paper's claims are 'comparable to SOTA' rather than 'new SOTA.' And the AIR-rotation loop has a structural circularity—you detect imbalance and then rotate to reduce that same imbalance—though the accuracy gains and noise experiments provide independent grounding.\n\nDon't over-penalize these. The authors are upfront about the single-layer theoretical scope and note that they don't directly improve unimodal encoders. The empirical work is reproducible in spirit, with code released.\n\nWho this is for: researchers in multimodal fusion, especially those working on attention-based dynamic fusion and modality imbalance. They will get a useful mechanism and a set of clear experiments. It is not a paradigm shift; it is a solid subfield contribution. My recommendation: send it to peer review. The referee should ask for a direct empirical check that average-key cosine similarity actually tracks softmax attention mass, plus error bars and a less hand-tuned story for the hyperparameters. But those are addressable issues. I'd bring it to our reading group and would cite it.","headline":"RollingQ is a solid empirical paper with a cheap fix for a real attention-bias problem, but its theoretical story conflates pre-softmax logits with attention mass, so treat the diagnosis as suggestive.","tokens_in":18791,"tokens_out":3173,"would_cite":true,"duration_ms":30647,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that dynamic attention fusion in multimodal Transformers collapses during training through a self-reinforcing bias loop, and that rotating the class-token query toward a rebalanced anchor restores it.","keywords":["multimodal learning","attention mechanism","dynamic fusion","modality bias","self-reinforcing cycle","query rotation","audio-visual learning","modality imbalance"],"falsifier":"Take a trained multimodal Transformer that exhibits the reported modality bias, compute the true softmax attention score for each modality, and compare it with the surrogate $L_m \\|q\\| \\|\\hat{k}_m\\| \\cos\\theta_m$ that RollingQ's diagnosis uses. If key norms or sequence lengths differ substantially between modalities, or if the surrogate ranks the modalities differently from the true attention on even a small fraction of held-out samples, the anchor and rotation target in Equations (9)-(13) are miscalibrated and RollingQ should be expected to fail.","tokens_in":17815,"feed_emoji":"🔄","tokens_out":11035,"duration_ms":92331,"temperature":0.7,"pith_summary":"Multimodal Transformers are supposed to fuse audio, visual, and text information by paying more attention to whichever modality is most informative for a given sample. The paper reports that this dynamic adaptability dies during training: the model locks onto one modality and keeps favoring it even when that modality is replaced by pure noise. The cause, it argues, is a self-reinforcing cycle in which the favored modality receives higher attention, its encoder receives more gradient, and the gap between modalities' attention-key distributions widens. The proposed fix, Rolling Query, rotates the class-token query toward a rebalancing anchor computed from the two modalities' average keys, interrupting the cycle. In experiments on audio-visual and vision-text benchmarks, this rotation restores attention's sensitivity to input quality and improves accuracy at negligible extra cost.","feed_headline":"Rotating the query revives dead attention in multimodal Transformers","feed_subtitle":"RollingQ breaks a self-reinforcing bias loop, restoring dynamic fusion and lifting accuracy on three benchmarks.","key_machinery":"The load-bearing object is the query rotation used by RollingQ. After each training epoch the method freezes the model, computes each modality's average key $\\hat{k}_m$, the Attention Imbalance Rate $\\mathrm{AIR} = \\mathbb{E}[\\cos\\theta_a - \\cos\\theta_v]$, and a balanced anchor $q_b = (\\alpha \\mathbb{E}[\\hat{K}_a]/\\lVert\\mathbb{E}[\\hat{K}_a]\\rVert_2 + (1-\\alpha)\\mathbb{E}[\\hat{K}_v]/\\lVert\\mathbb{E}[\\hat{K}_v]\\rVert_2)\\,\\lVert\\mathbb{E}[Q]\\rVert_2$ with $\\alpha = \\tfrac{1}{2}[1+\\mathrm{Tanh}(-\\rho\\,\\mathrm{AIR})]$. It then forms the rotation matrix $R_b = \\mathrm{SVD}([\\mathbb{E}[Q], q_b])$ satisfying $q_b = \\mathbb{E}[Q]R_b$ and replaces each query by $q_r = qR_b$ (Equation 13). The rotation preserves the query's norm, so it changes only which keys the query points toward; this redirects optimization momentum to the underused modality without adding a trainable module.","core_discovery":"The paper's central claim is that the loss of dynamic fusion in multimodal Transformers is a train-time pathology, not an architectural limitation. At initialization both modalities receive similar attention, but once one modality's features become slightly better, a feedback loop amplifies the advantage: feed-forward attention favors the biased modality, and backpropagation gives that modality's encoder more gradient, so its features improve further. The measurable signature of this loop is the Attention Imbalance Rate (AIR), the expected difference in cosine similarity between the query and the two modalities' average keys. RollingQ detects when the absolute AIR passes a threshold, builds a balanced anchor from the normalized average keys weighted by a Tanh function of AIR, and rotates the query by the singular-value-decomposition rotation matrix that maps the expected query onto the anchor. The rotated query then learns in a region where the previously neglected modality receives more attention, narrowing the key distribution gap and restoring the model's ability to lower attention when the biased modality is corrupted.","pith_inferences":["Because the AIR and anchor construction only use average keys and query expectations, the same rotation recipe should transfer to any attention-based fusion of two or more token sequences, such as vision-language or text-audio models, whenever key norms and sequence lengths are comparable.","Tracking AIR during training could serve as a standalone diagnostic for emerging modality collapse, since the paper's analysis links a widening gap to degraded adaptation even before accuracy suffers.","The rotation could plausibly be applied as a post-hoc correction on a frozen pretrained model, although the paper only reports train-time application; testing that variant would separate the training-dynamics effect from the inference-time redistribution.","Because RollingQ acts on the query rather than on encoder features, combining it with encoder-level rebalancing methods may yield further gains; the paper itself notes it does not directly equalize unimodal feature quality."],"forward_implications":["On the CREMA-D, Kinetic-Sound, and CMU-MOSEI benchmarks, RollingQ improves accuracy by 0.1 to 3.1 points over the corresponding vanilla or specialized multimodal Transformer baselines (Table 1).","When the biased modality is replaced with Gaussian noise, RollingQ sharply raises the correlation between attention scores and whether the input is noise (from about 0.44-0.52 to 0.76-0.78), showing that the attention mechanism has become sensitive to data quality again.","At every tested noise level, RollingQ suffers a smaller accuracy drop than the vanilla model, indicating better robustness to unreliable modalities.","The method adds roughly 1% parameters and 0.1% GFLOPs and improves performance across ViT depths, batch sizes, and a ResNet18 backbone, so the fix is cheap and portable."],"supporting_citations":[{"why":"defines the scaled dot-product self-attention formulation whose query-key-value structure the paper analyzes.","marker":"(Vaswani, 2017)"},{"why":"supplies the [class] token and its truncated-normal initialization, which anchors the attention-symmetry proposition at the start of training.","marker":"(Dosovitskiy, 2020)"},{"why":"provides the greedy multimodal learning premise that drives the self-reinforcing cycle.","marker":"(Wu et al., 2022)"},{"why":"supplies the singular value decomposition used to construct the query rotation matrix.","marker":"(Baker, 2005)"},{"why":"provides the Kinetic-Sound dataset where the attention-deactivation phenomenon is first observed.","marker":"(Arandjelovic & Zisserman, 2017)"},{"why":"provides the CREMA-D emotion-recognition benchmark used for the main accuracy and noise experiments.","marker":"(Cao et al., 2014)"},{"why":"provides the CMU-MOSEI dataset and the vision-plus-text setting used for generalization tests.","marker":"(Zadeh et al., 2018)"},{"why":"supplies the CMU-MOSEI preprocessing and the Gaussian-noise perturbation protocol used to test noisy modalities.","marker":"(Liang et al., 2021)"},{"why":"supplies the QUAG attention-impairment tests that support the claim that RollingQ strengthens inter-modal cooperation.","marker":"(Rawal et al., 2023)"},{"why":"provides the multimodal out-of-distribution benchmark setting used to test generalization under distribution shift.","marker":"(Dong et al., 2024)"}],"fun_headline_variants":["Rolling query breaks the attention bias snowball in Transformers","Rotating queries restores dynamic fusion in multimodal models","Simple query rotation revives cooperation between modalities","Bias loop in attention? RollingQ stops it before it snowballs","A rotation fix for self-reinforcing attention imbalance in Transformers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that a modality's attention score is essentially set by how similar the query is to that modality's average key, because the keys' sizes and sequence lengths are roughly equal across the two modalities.","fun_headline_variants_meta":{"raw":{"variants":["Rolling query breaks the attention bias snowball in Transformers","Rotating queries restores dynamic fusion in multimodal models","Simple query rotation revives cooperation between modalities","Bias loop in attention? RollingQ stops it before it snowballs","A rotation fix for self-reinforcing attention imbalance in Transformers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00034,"raw_usage":{"total_tokens":1869,"prompt_tokens":935,"completion_tokens":934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":851}},"tokens_in":551,"tokens_out":934,"duration_ms":9797,"temperature":1.0,"reasoning_tokens":851,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:45.405241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained multimodal Transformer that exhibits the reported modality bias, compute the true softmax attention score for each modality, and compare it with the surrogate $L_m \\|q\\| \\|\\hat{k}_m\\| \\cos\\theta_m$ that RollingQ's diagnosis uses. If key norms or sequence lengths differ substantially between modalities, or if the surrogate ranks the modalities differently from the true attention on even a small fraction of held-out samples, the anchor and rotation target in Equations (9)-(13) are miscalibrated and RollingQ should be expected to fail.","supporting_citations":[{"cited_title":"Singular value decomposition tutorial","cited_arxiv_id":null,"evidence_quote":"supplies the singular value decomposition used to construct the query rotation matrix."},{"cited_title":"and Zisserman, A","cited_arxiv_id":null,"evidence_quote":"provides the Kinetic-Sound dataset where the attention-deactivation phenomenon is first observed."},{"cited_title":"G., Keutmann, M","cited_arxiv_id":null,"evidence_quote":"provides the CREMA-D emotion-recognition benchmark used for the main accuracy and noise experiments."},{"cited_title":"B., Liang, P","cited_arxiv_id":null,"evidence_quote":"provides the CMU-MOSEI dataset and the vision-plus-text setting used for generalization tests."},{"cited_title":"P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M","cited_arxiv_id":null,"evidence_quote":"supplies the CMU-MOSEI preprocessing and the Gaussian-noise perturbation protocol used to test noisy modalities."},{"cited_title":"MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities","cited_arxiv_id":"2405.17419","evidence_quote":"provides the multimodal out-of-distribution benchmark setting used to test generalization under distribution shift."}],"review_version":1}