Pith. sign in

REVIEW 4 major objections 5 minor 50 references

This paper claims that aligning EEG signals with both images and their text labels, then dynamically rebalancing each modality's gradient contribution, lifts EEG-based visual decoding on ThingsEEG to 15.8% Top-1 and 44.2% Top-5 accuracy—bea

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A multimodal EEG-image-text contrastive framework with dynamic gradient balancing and stochastic noise improves zero-shot object recognition from EEG on ThingsEEG, raising top-1 accuracy from 13.8% to 15.8%.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Internally inconsistent results undermine the SOTA claim, but the method is a reasonable extension worth a referee after reconciliation. the 4 major comments →

arxiv 2509.03433 v1 pith:CF7I67XR submitted 2025-09-03 cs.CV

Decoding Visual Neural Representations by Multimodal with Dynamic Balancing

Kaili sun , Xingyu Miao , Bing Zhai , Haoran Duan , Yang Long This is my paper
classification cs.CV
keywords brain neural decodingEEGmultimodal contrastive learninggradient modulationmodal balancevisual decodingzero-shot classificationThingsEEG
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that EEG-based visual decoding—recognizing which natural image a person is viewing from their brain signals—can be substantially improved by aligning EEG features not just with images but also with text labels of the same categories. The core idea is that text supplies explicit semantic anchors that pull noisy EEG representations and image features closer together in a shared space. Because text features are high-dimensional and tend to dominate joint training, the paper adds a dynamic balancing mechanism (MCDB) that measures each modality's contribution from softmax similarity matrices and scales down the dominant modality's gradient updates, plus a stochastic perturbation regularization (SPR) that injects decaying Gaussian noise to escape local optima. On ThingsEEG across ten subjects, the full framework reaches 15.8% Top-1 and 44.2% Top-5 accuracy with 63 channels, beating BraVL by 10.0/26.7 points and NICE by 2.0/4.7 points. If this holds, text-grounded multimodal alignment plus gradient balancing is a viable route to more accurate zero-shot decoding of visual content from EEG.

Core claim

On the paper's own terms, the central discovery is that the dominant reason EEG-based visual decoding falls short is not only the noise of EEG but an optimization imbalance between modalities in shared-space contrastive learning, and that this imbalance can be corrected cheaply. Adding text as a third modality makes the shared feature distribution tighter—EEG and image features of the same category are pulled toward the same text anchor—but the text modality then dominates gradients in later training. The framework's Modal Consistency Dynamic Balance (MCDB) measures dominance by the column sum of the softmax similarity matrix between EEG and each other modality, forms the imbalance rate rho(

What carries the argument

The load-bearing mechanism is a three-part optimization stack rather than a single identity: (1) a text-as-semantic-bridge loss (four cross-entropy contrastive terms plus image-text cosine loss) that makes text labels the anchor for EEG and image alignment; (2) MCDB, a gradient-modulation rule built on the column sums of the softmax similarity matrix S^{(eeg,m)} = softmax(F^{(eeg)}(F^{(m)})^T/tau); the imbalance rate rho(m) = ||s(eeg,m)||_1/(sum_{n≠m}||s(eeg,n)||_1+epsilon) determines kappa(m)=1-tanh(gamma(rho(m)-1)) for rho(m)>1, so the dominant modality's gradient is damped and others are left untouched; and (3) SPR, which injects Gaussian noise h(theta_t) ~ N(0, sigma_t^2 I) with sigma_t

Load-bearing premise

MCDB assumes the column sum of the softmax similarity between EEG and a modality faithfully measures how much that modality dominates the shared space, so scaling its gradient by kappa(m) improves alignment rather than merely slowing learning.

What would settle it

Run the same framework with MCDB disabled (kappa fixed to 1) while keeping text, adapter, and SPR; if Top-1/Top-5 do not fall well below 15.8/44.2, the balance mechanism is not the source of the gain. Because sigma_max and beta are not reported, also re-run across a sweep of noise schedules; if final accuracy is insensitive to the noise schedule or collapses without it, the SPR claim needs revision.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Text modality should be included in EEG-image contrastive pipelines since it provides explicit semantic anchors that tighten the shared feature distribution.
  • Modality dominance can be fixed at the optimizer level (gradient scaling) without training auxiliary networks, so the balancing cost is only added gradient computation.
  • The 17-channel subset gives nearly the same Top-5 accuracy as 63 channels (42.3% vs 44.2%), suggesting visual decoding may be possible with fewer electrodes.
  • Gradient trajectories after modulation—text gradients stabilize, image gradients rise, EEG gradients fluctuate less—are the claimed evidence that balancing improves collaborative learning across subjects.
  • The time-window and cortical ablation analyses support a neural interpretation: occipital electrodes carry the most information (removal drops accuracy 4.6%), and the 100-600ms window is the most informative.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The MCDB dominance proxy (column-sum softmax similarity) is never validated independently of final accuracy; if it is miscalibrated, the same mechanism could suppress useful signal in other settings—an empirical question the paper leaves open.
  • Because text anchors are categorical labels and descriptions, the framework is most natural for categories with rich language supervision; for purely perceptual categories with no linguistic labels, the text bridge may add little.
  • The paper does not report the chosen values of sigma_max and beta for SPR, so the noise schedule is underspecified; a useful extension is to sweep these parameters per subject and test whether optimal noise scales with EEG signal-to-noise ratio.
  • If the balancing approach transfers, it applies beyond brain decoding: any multimodal contrastive learner with one dominant modality could use the same kappa(m) gradient rule, though the paper only demonstrates it on EEG-image-text.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HMA-VD, a multimodal EEG decoding framework that combines EEG, image, and text with CLIP encoders, bottleneck adapters, the MCDB gradient-balancing mechanism, and SPR stochastic noise injection. The central empirical claim is that on the ThingsEEG dataset with 63 channels, HMA-VD achieves 15.8% Top-1 / 44.2% Top-5 average accuracy, improving over NICE by 2.0% / 4.7%. The paper also reports 17-channel results, ablations, gradient analyses, and a biological rationale analysis. The method is an engineering combination of established contrastive losses and gradient modulation, and the main claimed contribution is the observed accuracy gain.

Significance. If the reported result is reproducible and the hyperparameters are honestly selected, the paper makes a moderate but useful empirical contribution: it shows that text-bridged alignment plus dynamic gradient balancing can improve EEG-to-image zero-shot classification on the largest public benchmark of this kind. The paper has several strengths: per-subject results are reported, comparisons to BraVL and NICE are included, ablation and hyperparameter sweeps are attempted, and the gradient analysis provides some qualitative support for the balancing mechanism. However, the central claim is currently undermined by an internal inconsistency between the main results table and the ablation table, by missing values for key noise/fusion hyperparameters, and by an unclear hyperparameter-selection protocol. These issues prevent the results from being independently verified as stated.

major comments (4)
  1. [Section 4.4 vs Section 4.7, Tables 2 and 3] The headline result in Table 2 is HMA-VD-63 average 15.8/44.2. The full-model row of Table 3 (Base+Text+Adapter+MCDB+SPR), described under the same controlled settings, reports 15.2/41.9. This is a gap of 0.6 Top-1 and 2.3 Top-5, for which the paper offers no explanation. If the ablation row is the correct number, the claimed improvement over NICE shrinks from +2.0/+4.7 to +1.4/+2.4. If Table 2 is correct, then the ablation protocol or checkpoint selection differs in an undisclosed way. Additionally, Table 3 row 2 is nearly identical to Table 2 HMA-VD-17 but with Sub5 Top-5 reported as 27.0 instead of 32.5, another unexplained discrepancy. Because no code, seeds, or raw per-subject outputs are provided, a reader cannot determine which value reflects the method.
  2. [Section 4.2 and Eqs. (8), (17)] Several hyperparameters that control the proposed mechanisms are never reported. Section 4.2 lists alpha=0.7, tau=0.07, Adam betas, and learning rate, but the fusion weight beta in Eq. (8) and the SPR parameters sigma_max and the decay exponent beta in Eq. (17) are absent from the experiments. The ablation tables only sweep gamma and the text/image compression ratio. Without these values, the SPR behavior is uncontrolled and the method cannot be reproduced from the paper. The authors should provide exact values for all free parameters, and preferably sensitivity analyses for sigma_max and beta.
  3. [Section 4.7, Tables 4 and 5] Tables 4 and 5 are presented as ablations, but the paper does not state whether the hyperparameter values gamma and r were selected on the validation split or on the test set. Notably, the reported best settings gamma=0.7 and r=16/8 exactly reproduce the Table 2 average of 15.8/44.2. Section 4.2 only describes using the validation set to select the checkpoint, not to tune hyperparameters. If gamma and r were chosen after inspecting test-set performance, the reported gains over NICE are optimistically biased. The paper should state the model-selection protocol explicitly and, if test-based selection occurred, re-evaluate on a held-out split.
  4. [Section 3.3, Eqs. (10)-(13)] The MCDB mechanism assumes that the column sum of the softmax similarity matrix, s^(eeg,m), is a faithful proxy for modality contribution, and that scaling gradients by kappa(m) helps joint optimization. No supporting analysis is provided beyond final accuracy and qualitative gradient curves. Also, Eq. (12) defines rho(m) as the ratio of one modality's contribution to the sum of the other two; with three modalities a modality can be 'dominant' in the sense of having a plurality (e.g., 40% vs 30%/30%) without ever exceeding 1, so the suppressor would never activate. The paper should either justify the proxy theoretically, show quantitative rho/kappa trajectories, or compare MCDB against an existing gradient-balancing baseline such as OGM to demonstrate that the mechanism behaves as intended.
minor comments (5)
  1. [Eq. (7) and Eq. (2)] Eq. (7) defines L_r as a cosine similarity and calls it a loss, but maximizing cosine similarity requires minimizing -L_r or equivalently 1-L_r. Also, Eq. (2) does not include L_r in the overall loss L; the paper should clarify how the adapter loss is optimized and how it is combined with the other terms.
  2. [Table 2] The table uses the abbreviations SA and GA without defining them. The NICE-SA/NICE-GA variants should be explained the first time they appear, or at least cited to the relevant equations in [13].
  3. [Section 4.2] The EEG encoder is only described as 'TSconv'. The paper should specify the architecture or cite the exact implementation, since the encoder is a central component and the comparison with NICE depends on it.
  4. [Section 4.3, Table 1] The text says 'the computational efficiency of our method is reduced' but Table 1 reports a relative time of 1.59 versus NICE. The wording should be clearer: the method is slower in training but the paper claims this is an acceptable trade-off.
  5. [Author contributions] The author contributions list names 'Yang Bai' in the writing/review role, but Yang Bai is not listed as an author. This should be corrected.

Circularity Check

0 steps flagged

No significant circularity: HMA VD is an empirical pipeline evaluated against external baselines; self-citations are not load-bearing.

full rationale

I traced the paper's derivation chain. The central claim is an accuracy improvement over BraVL/NICE, computed directly from Table 2 as the difference between reported averages; it is not derived from the model equations themselves. The loss functions in Eq. 2-4, the adapter in Eq. 5-8, and the MCDB/SPR gradient mechanism in Eq. 9-20 are training objectives and optimizer modifications. MCDB's rho(m) is computed from current-batch similarity matrices and used to scale gradients; this is a self-referential training heuristic, but not a logical circle, because it does not define the target accuracy in terms of itself. SPR introduces Gaussian noise with hyperparameters sigma_max and beta that are not reported, which is a reproducibility gap but not circularity. The paper cites several of the authors' prior works (e.g., [4,5,23,24,25,28,34,35]), but these are confined to related-work context and are not used to justify the method's validity or to forbid alternative approaches. No uniqueness theorem is imported from the authors. The internal inconsistency between Table 2's full-model result (15.8/44.2) and Table 3's full-model row (15.2/41.9) is a reproducibility/correctness concern that the authors should address, but it is not an instance of a prediction reducing to a fitted input or of a claim being equivalent to its inputs by construction. Overall, the method is a standard empirical pipeline with external baselines, and I find no significant circularity.

Axiom & Free-Parameter Ledger

9 free parameters · 5 axioms · 0 invented entities

The method rests on CLIP's pretrained space, on the assumption that EEG contains usable category signal, and on several unvalidated modeling choices (MCDB proxy, SPR noise). It does not introduce new physical entities, but the numerous undisclosed hyperparameters increase the burden on reimplementation.

free parameters (9)
  • alpha (adapter residual scaling) = 0.7
    Fixed in Section 4.2; controls residual strength in adapters (Eq. 5-6).
  • gamma (MCDB sensitivity) = 0.7
    Tuned on validation set; Table 4 shows best at 0.7.
  • r_text, r_img (adapter compression ratios) = 16, 8
    Chosen empirically (Table 5) to balance dimensionality reduction and information retention.
  • beta_fusion (adapter fusion weight) = not reported
    Eq. 8 uses beta to fuse original and adapter text features; value or learning rule is not disclosed.
  • sigma_max (SPR initial noise) = not reported
    Eq. 17 defines noise schedule but sigma_max is not specified.
  • beta_noise (SPR decay exponent) = not reported
    Decay exponent in Eq. 17; not reported in experiments.
  • tau (temperature) = 0.07
    Set to CLIP default (Section 4.2).
  • Adam beta1/beta2 = 0.5, 0.999
    Section 4.2; beta1=0.5 is unusual and differs from standard Adam default.
  • loss term weight = 1/5 each
    Equation 2 averages five loss terms equally; no justification for equal weighting.
axioms (5)
  • domain assumption CLIP's pretrained image-text space provides a valid semantic anchor for aligning EEG features
    Section 3.1 uses text features from CLIP as a bridge; if CLIP's space does not match neural category structure, the bridging fails.
  • domain assumption ThingsEEG preprocessing (trial averaging, multivariate noise normalization) preserves sufficient category information
    Section 4.1 describes preprocessing; the method's success depends on these signals being class-discriminative.
  • ad hoc to paper The softmax column-sum similarity (Eq. 10-12) is a faithful proxy for modality contribution
    The MCDB imbalance rate is a novel heuristic; no proof or ablation validates the proxy itself.
  • domain assumption Adding Gaussian noise to gradients improves generalization without breaking convergence
    Section 3.3 motivates SPR via stochastic resonance; the schedule and magnitude are not justified empirically.
  • standard math Equal-weight combination of four cross-entropy losses and one cosine loss is a suitable objective
    Equation 2 defines the training loss; equal weighting is a design choice.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoding Visual Neural Representations by Multimodal with Dynamic Balancing." pith.science (2026). https://pith.science/paper/CF7I67XR

@misc{pith2026250903433,
  author       = {Pith},
  title        = {Pith review of: Decoding Visual Neural Representations by Multimodal with Dynamic Balancing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CF7I67XR}},
  note         = {Machine review of arXiv:2509.03433}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In this work, we propose an innovative framework that integrates EEG, image, and text data, aiming to decode visual neural representations from low signal-to-noise ratio EEG signals. Specifically, we introduce text modality to enhance the semantic correspondence between EEG signals and visual content. With the explicit semantic labels provided by text, image and EEG features of the same category can be more closely aligned with the corresponding text representations in a shared multimodal space. To fully utilize pre-trained visual and textual representations, we propose an adapter module that alleviates the instability of high-dimensional representation while facilitating the alignment and fusion of cross-modal features. Additionally, to alleviate the imbalance in multimodal feature contributions introduced by the textual representations, we propose a Modal Consistency Dynamic Balance (MCDB) strategy that dynamically adjusts the contribution weights of each modality. We further propose a stochastic perturbation regularization (SPR) term to enhance the generalization ability of semantic perturbation-based models by introducing dynamic Gaussian noise in the modality optimization process. The evaluation results on the ThingsEEG dataset show that our method surpasses previous state-of-the-art methods in both Top-1 and Top-5 accuracy metrics, improving by 2.0\% and 4.7\% respectively.

Figures

Figures reproduced from arXiv: 2509.03433 by Bing Zhai, Haoran Duan, Kaili Sun, Xingyu Miao, Yang Long.

Figure 1
Figure 1. Figure 1: Dual-channel integration of knowledge in the human brain: When the human eye [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall Framework for multimodal EEG-Image-Text Alignment: During train [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (A) EEG and image feature distribution in a shared space. The distribution is [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Structure of the adapter layer (right). The adapter layer consists of two linear [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Effects of dynamic gradient modulation on EEG optimization: pre-modulation [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Dynamic Cortical Activity Topographies Averaged Over 100 ms Intervals: Initial [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (A) The average accuracy of 10 subjects in different time periods, with the 100- [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: (A) Gradient norm trajectories for Subject 1 before and after modulation. The [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 40 canonical work pages · 3 internal anchors

  1. [1]

    Gait adaptation to visual kinematic perturbations using a real-time closed-loop brain–computer interface to a virtual reality avatar,

    T. P. Luu, Y. He, S. Brown, S. Nakagome, and J. L. Contreras-Vidal, “Gait adaptation to visual kinematic perturbations using a real-time closed-loop brain–computer interface to a virtual reality avatar,”Journal of neural engineering , vol. 13, no. 3, p. 036006, 2016. 29

    work page 2016

  2. [2]

    An online ssvep-bci system in an optical see-through augmented reality environment,

    Y. Ke, P. Liu, X. An, X. Song, and D. Ming, “An online ssvep-bci system in an optical see-through augmented reality environment,” Journal of neural engineering, vol. 17, no. 1, p. 016066, 2020

    work page 2020

  3. [3]

    Repvgg: Mak- ing vgg-style convnets great again,

    X. Ding, X. Zhang, N. Ma, J. Han, G. Ding, and J. Sun, “Repvgg: Mak- ing vgg-style convnets great again,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 13 733– 13 742

    work page 2021

  4. [4]

    Rethinking score distilling sampling for 3d editing and generation,

    X. Miao, H. Duan, Y. Long, and J. Han, “Rethinking score distilling sampling for 3d editing and generation,” arXiv preprint arXiv:2505.01888, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Dreamer xl: Towards high-resolution text-to-3d generation via trajec- tory score matching,

    X. Miao, H. Duan, V. Ojha, J. Song, T. Shah, Y. Long, and R. Ranjan, “Dreamer xl: Towards high-resolution text-to-3d generation via trajec- tory score matching,” arXiv preprint arXiv:2405.11252 , 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Decoding individual finger movements from one hand using human eeg signals,

    K. Liao, R. Xiao, J. Gonzalez, and L. Ding, “Decoding individual finger movements from one hand using human eeg signals,” PloS one , vol. 9, no. 1, p. e85192, 2014

    work page 2014

  7. [7]

    Eeg-based bci control schemes for lower-limb assistive-robots,

    M. Tariq, P. M. Trivailo, and M. Simic, “Eeg-based bci control schemes for lower-limb assistive-robots,” Frontiers in human neuro- science, vol. 12, p. 312, 2018

    work page 2018

  8. [8]

    Deep learning for elec- troencephalogram (eeg) classification tasks: a review,

    A. Craik, Y. He, and J. L. Contreras-Vidal, “Deep learning for elec- troencephalogram (eeg) classification tasks: a review,” Journal of neural engineering, vol. 16, no. 3, p. 031001, 2019

    work page 2019

  9. [9]

    Real-time neuroimaging and cognitive monitoring using wearable dry eeg,

    T. R. Mullen, C. A. Kothe, Y. M. Chi, A. Ojeda, T. Kerth, S. Makeig, T.- P. Jung, and G. Cauwenberghs, “Real-time neuroimaging and cognitive monitoring using wearable dry eeg,” IEEE transactions on biomedical engineering, vol. 62, no. 11, pp. 2553–2567, 2015

    work page 2015

  10. [10]

    Making sense of spatio-temporal preserving representations for eeg-based human intention recognition,

    D. Zhang, L. Yao, K. Chen, S. Wang, X. Chang, and Y. Liu, “Making sense of spatio-temporal preserving representations for eeg-based human intention recognition,” IEEE transactions on cybernetics , vol. 50, no. 7, pp. 3033–3044, 2019

    work page 2019

  11. [11]

    Deep learning with convolutional neural networks for eeg de- coding and visualization,

    R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstet- ter, K. Eggensperger, M. Tangermann, F. Hutter, W. Burgard, and 30 T. Ball, “Deep learning with convolutional neural networks for eeg de- coding and visualization,” Human brain mapping , vol. 38, no. 11, pp. 5391–5420, 2017

    work page 2017

  12. [12]

    Eeg conformer: Convolutional transformer for eeg decoding and visualization,

    Y. Song, Q. Zheng, B. Liu, and X. Gao, “Eeg conformer: Convolutional transformer for eeg decoding and visualization,” IEEE Transactions on Neural Systems and Rehabilitation Engineering , vol. 31, pp. 710–719, 2022

    work page 2022

  13. [13]

    Decod- ing natural images from eeg for object recognition,

    Y. Song, B. Liu, X. Li, N. Shi, Y. Wang, and X. Gao, “Decod- ing natural images from eeg for object recognition,” arXiv preprint arXiv:2308.13234, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Eeg-based emotion recognition with haptic vibration by a feature fu- sion method,

    D. Li, Z. Yang, F. Hou, Q. Kang, S. Liu, Y. Song, Q. Gao, and E. Dong, “Eeg-based emotion recognition with haptic vibration by a feature fu- sion method,” IEEE Transactions on Instrumentation and Measure- ment, vol. 71, pp. 1–11, 2022

    work page 2022

  15. [15]

    Dewave: Discrete eeg waves encoding for brain dynamics to text translation,

    Y. Duan, J. Zhou, Z. Wang, Y.-K. Wang, and C.-T. Lin, “Dewave: Discrete eeg waves encoding for brain dynamics to text translation,” arXiv preprint arXiv:2309.14030 , 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Paivio, Imagery and verbal processes

    A. Paivio, Imagery and verbal processes . Psychology Press, 2013

    work page 2013

  17. [17]

    Dual coding of knowledge in the human brain,

    Y. Bi, “Dual coding of knowledge in the human brain,” Trends in Cog- nitive Sciences, vol. 25, no. 10, pp. 883–895, 2021

    work page 2021

  18. [18]

    Decoding visual neural representations by multimodal learning of brain-visual-linguistic features,

    C. Du, K. Fu, J. Li, and H. He, “Decoding visual neural representations by multimodal learning of brain-visual-linguistic features,” IEEE Trans- actions on Pattern Analysis and Machine Intelligence , vol. 45, no. 9, pp. 10 760–10 777, 2023

    work page 2023

  19. [19]

    What makes training multi-modal classification networks hard?

    W. Wang, D. Tran, and M. Feiszli, “What makes training multi-modal classification networks hard?” in Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , 2020, pp. 12 695– 12 705

    work page 2020

  20. [20]

    Balanced multi- modal learning via on-the-fly gradient modulation,

    X. Peng, Y. Wei, A. Deng, D. Wang, and D. Hu, “Balanced multi- modal learning via on-the-fly gradient modulation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 8238–8247. 31

    work page 2022

  21. [21]

    Im- proving multi-modal learning with uni-modal teachers,

    C. Du, T. Li, Y. Liu, Z. Wen, T. Hua, Y. Wang, and H. Zhao, “Im- proving multi-modal learning with uni-modal teachers,” arXiv preprint arXiv:2106.11059, 2021

    Pith/arXiv arXiv 2021

  22. [22]

    Uni-moe: Scaling unified multimodal llms with mixture of experts,

    Y. Li, S. Jiang, B. Hu, L. Wang, W. Zhong, W. Luo, L. Ma, and M. Zhang, “Uni-moe: Scaling unified multimodal llms with mixture of experts,” arXiv preprint arXiv:2405.11273 , 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Dynamic unary convolution in transformers,

    H. Duan, Y. Long, S. Wang, H. Zhang, C. G. Willcocks, and L. Shao, “Dynamic unary convolution in transformers,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 11, pp. 12 747– 12 759, 2023

    work page 2023

  24. [24]

    Pa- rameter efficient fine-tuning for multi-modal generative vision models with m¨ obius-inspired transformation,

    H. Duan, S. Shao, B. Zhai, T. Shah, J. Han, and R. Ranjan, “Pa- rameter efficient fine-tuning for multi-modal generative vision models with m¨ obius-inspired transformation,”International Journal of Com- puter Vision , vol. 133, no. 7, pp. 4590–4603, 2025

    work page 2025

  25. [25]

    Ctnerf: Cross-time transformer for dynamic neural radiance field from monocular video,

    X. Miao, Y. Bai, H. Duan, F. Wan, Y. Huang, Y. Long, and Y. Zheng, “Ctnerf: Cross-time transformer for dynamic neural radiance field from monocular video,” Pattern Recognition, vol. 156, p. 110729, 2024

    work page 2024

  26. [26]

    The benefits of noise in neural sys- tems: bridging theory and experiment,

    M. D. McDonnell and L. M. Ward, “The benefits of noise in neural sys- tems: bridging theory and experiment,” Nature Reviews Neuroscience, vol. 12, no. 7, pp. 415–425, 2011

    work page 2011

  27. [27]

    Improving unimodal object recognition with multimodal contrastive learning,

    J. Meyer, A. Eitel, T. Brox, and W. Burgard, “Improving unimodal object recognition with multimodal contrastive learning,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 5656–5663

    work page 2020

  28. [28]

    Ds-depth: Dynamic and static depth estimation via a fusion cost volume,

    X. Miao, Y. Bai, H. Duan, Y. Huang, F. Wan, X. Xu, Y. Long, and Y. Zheng, “Ds-depth: Dynamic and static depth estimation via a fusion cost volume,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 4, pp. 2564–2576, 2023

    work page 2023

  29. [29]

    Mitigating Propensity Bias of Large Language Models for Recommender Systems

    G. Zhang, G. Yuan, D. Cheng, L. Liu, J. Li, and S. Zhang, “Mitigating propensity bias of large language models for recommender systems,” arXiv preprint arXiv:2409.20052 , 2024. 32

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [30]

    Bayesian graph local extrema con- volution with long-tail strategy for misinformation detection,

    G. Zhang, S. Zhang, and G. Yuan, “Bayesian graph local extrema con- volution with long-tail strategy for misinformation detection,” ACM Transactions on Knowledge Discovery from Data , vol. 18, no. 4, pp. 1–21, 2024

    work page 2024

  31. [31]

    Multimodal contrastive training for visual representation learning,

    X. Yuan, Z. Lin, J. Kuen, J. Zhang, Y. Wang, M. Maire, A. Kale, and B. Faieta, “Multimodal contrastive training for visual representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 6995–7004

    work page 2021

  32. [32]

    Un- derstanding multimodal contrastive learning and incorporating unpaired data,

    R. Nakada, H. I. Gulluk, Z. Deng, W. Ji, J. Zou, and L. Zhang, “Un- derstanding multimodal contrastive learning and incorporating unpaired data,” in International Conference on Artificial Intelligence and Statis- tics. PMLR, 2023, pp. 4348–4380

    work page 2023

  33. [33]

    Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning,

    H. Bansal, N. Singhi, Y. Yang, F. Yin, A. Grover, and K.-W. Chang, “Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 112–123

    work page 2023

  34. [34]

    Laser: Efficient language-guided segmentation in neural radiance fields,

    X. Miao, H. Duan, Y. Bai, T. Shah, J. Song, Y. Long, R. Ranjan, and L. Shao, “Laser: Efficient language-guided segmentation in neural radiance fields,” arXiv preprint arXiv:2501.19084 , 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting

    X. Miao, H. Duan, Q. Qian, J. Wang, Y. Long, L. Shao, D. Zhao, R. Xu, and G. Zhang, “Towards scalable spatial intelligence via 2d-to-3d data lifting,” arXiv preprint arXiv:2507.18678 , 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    work page 2021

  37. [37]

    Wav2clip: Learning robust audio representations from clip,

    H.-H. Wu, P. Seetharaman, K. Kumar, and J. P. Bello, “Wav2clip: Learning robust audio representations from clip,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP) . IEEE, 2022, pp. 4563–4567

    work page 2022

  38. [38]

    Multimodal- ity helps unimodality: Cross-modal few-shot learning with multimodal 33 models,

    Z. Lin, S. Yu, Z. Kuang, D. Pathak, and D. Ramanan, “Multimodal- ity helps unimodality: Cross-modal few-shot learning with multimodal 33 models,” in Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , 2023, pp. 19 325–19 337

    work page 2023

  39. [39]

    Ulip-2: To- wards scalable multimodal pre-training for 3d understanding,

    L. Xue, N. Yu, S. Zhang, A. Panagopoulou, J. Li, R. Mart ´ ın-Mart ´ ın, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, “Ulip-2: To- wards scalable multimodal pre-training for 3d understanding,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 27 091–27 101

    work page 2024

  40. [40]

    See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity

    Z. Ye, L. Yao, Y. Zhang, and S. Gustin, “See what you see: Self- supervised cross-modal retrieval of visual stimuli from brain activity,” arXiv preprint arXiv:2208.03666 , 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  41. [41]

    Mind’s eye: Image recognition by eeg via multimodal similarity-keeping contrastive learning,

    C.-S. Chen and C.-S. Wei, “Mind’s eye: Image recognition by eeg via multimodal similarity-keeping contrastive learning,” arXiv preprint arXiv:2406.16910, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Dreamd- iffusion: Generating high-quality images from brain eeg signals,

    Y. Bai, X. Wang, Y.-p. Cao, Y. Ge, C. Yuan, and Y. Shan, “Dreamd- iffusion: Generating high-quality images from brain eeg signals,” arXiv preprint arXiv:2306.16934, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Mindbridge: A cross-subject brain decoding framework,

    S. Wang, S. Liu, Z. Tan, and X. Wang, “Mindbridge: A cross-subject brain decoding framework,” in Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , 2024, pp. 11 333– 11 342

    work page 2024

  44. [44]

    Wills aligner: Multi-subject collaborative brain visual decoding,

    G. Bao, Q. Zhang, Z. Gong, J. Zhou, W. Fan, K. Yi, U. Naseem, L. Hu, and D. Miao, “Wills aligner: Multi-subject collaborative brain visual decoding,” in Proceedings of the AAAI Conference on Artificial Intelli- gence, vol. 39, no. 13, 2025, pp. 14 194–14 202

    work page 2025

  45. [45]

    Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction,

    Z. Gong, Q. Zhang, G. Bao, L. Zhu, R. Xu, K. Liu, L. Hu, and D. Miao, “Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction,” in Proceedings of the AAAI Conference on Artifi- cial Intelligence , vol. 39, no. 13, 2025, pp. 14 247–14 255

    work page 2025

  46. [46]

    Gnn-film: Graph neural networks with feature-wise linear modulation,

    M. Brockschmidt, “Gnn-film: Graph neural networks with feature-wise linear modulation,” in International Conference on Machine Learning . PMLR, 2020, pp. 1144–1152. 34

    work page 2020

  47. [47]

    Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis,

    W. Han, H. Chen, and S. Poria, “Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis,” arXiv preprint arXiv:2109.00412 , 2021

    Pith/arXiv arXiv 2021

  48. [48]

    Boosting multi-modal model performance with adaptive gradient modulation,

    H. Li, X. Li, P. Hu, Y. Lei, C. Li, and Y. Zhou, “Boosting multi-modal model performance with adaptive gradient modulation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 22 214–22 224

    work page 2023

  49. [49]

    Suppress and rebalance: Towards generalized multi-modal face anti- spoofing,

    X. Lin, S. Wang, R. Cai, Y. Liu, Y. Fu, W. Tang, Z. Yu, and A. Kot, “Suppress and rebalance: Towards generalized multi-modal face anti- spoofing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 211–221

    work page 2024

  50. [50]

    Enhancing multimodal co- operation via sample-level modality valuation,

    Y. Wei, R. Feng, Z. Wang, and D. Hu, “Enhancing multimodal co- operation via sample-level modality valuation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 27 338–27 347. 35

    work page 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.