Pith. sign in

REVIEW 4 major objections 5 minor 51 references

MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulation

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A dual-level gradient modulation can make multimodal medical classifiers both more accurate and more equitable.

desk verdict The empirical combination is real and worth a look, but the central group-level modulation mechanism does not follow from the equations as written. read the letter →

arxiv 2510.07328 v2 pith:U44B4QRI submitted 2025-09-30 cs.LG cs.AIcs.CVcs.CY

Md Zubair , Hao Zheng , Grayson W. Armstrong , Lucy Q. Shen , Gabriela Wilson , Yu Tian , Xingquan Zhu This is my paper
classification cs.LGcs.AIcs.CVcs.CY
keywords multimodallearningfairnessgradientmodulationmedicalclassificationglaucomaAUCdemographicgroupsES-AUC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MultiFair claims that multimodal medical classifiers can be trained to be both more accurate and fairer by modulating gradients at two levels simultaneously. At the modality level, it slows down fast-learning modalities and boosts slow ones using per-modality AUC improvements. At the group level, it adds a fairness loss computed from per-group, per-modality exponential-moving-average AUC surrogates, and scales the encoder gradient by a fairness factor built from group performance gaps. On the FairVision and FairCLIP glaucoma datasets, the paper reports that MultiFair outperforms existing unimodal, fairness-aware, and balanced multimodal baselines on both AUC and equity-scaled AUC, with gains coming primarily from lifting underrepresented groups. If the result holds, it would mean fairness can be improved in multimodal medical diagnosis without the usual accuracy trade-off.

What carries the argument

The load-bearing mechanism is the dual-level gradient modulation: a modality balancing factor B_i = ρ·(Σ_{k≠i} ΔAUC_k)/(Σ_k ΔAUC_k) that boosts slow-learning encoders; a gradient-direction alignment loss L_gm that penalizes cosine misalignment between encoder and fusion gradients; a group-modality fairness factor F_g^i = 1 + δ·(AUC_EMA_i − AUC_EMA_gi)/τ; a batch-level aggregation f_batch_i = Σ_g p_g F_g^i; and a fairness loss F_G = (1/M)Σ_i(1/G)Σ_g|AUC_EMA_gi − AUC_EMA_i|. These are combined as L_total = L_task + λ_gm L_gm + λ_f F_G, with encoder updates θ ← θ − α B_i f_batch_i ∇L_total and fairness modulation triggered only when the fusion model's group AUC gap exceeds threshold τ.

What would settle it

Train the same architecture twice on FairVision: once with the full MultiFair loss, once with f_batch_i set to 1 (i.e., no fairness scaling) while keeping F_G in the loss. If per-group AUC trajectories change little between the two runs, the scalar f_batch is not doing the group-level work the paper credits to it. Conversely, if removing only F_G eliminates group-level improvements, the fairness loss is the true mechanism. Either outcome is directly observable from the paper's tables by comparing these two ablated runs.

Watch

Extended reading notes

Core claim

The central claim is that modality imbalance and demographic unfairness are entangled in multimodal medical learning, and that a unified training-time gradient modulation can fix both at once. MultiFair couples a classifier-guided modality balancing factor (derived from each modality's AUC improvement) with a group-level fairness factor (derived from per-group EMA AUC deviations) and a fairness penalty into a single loss. The paper argues this lets the model converge to a stationary point where both the modality balance loss and the fairness gap shrink monotonically, and empirically shows on two glaucoma datasets that the combined mechanism beats the best unimodal, fairness-only, and balance

Load-bearing premise

The paper assumes that a single scalar multiplier computed from group-AUC gaps and applied to the whole encoder gradient can 'emphasize updating the respective group'; since the same scale applies to every sample in the modality, group-selective learning can only arise through gradients of the fairness loss F_G, and that pathway is never shown.

Editorial extensions

If this is right

  • If the central claim is correct, clinicians can retrofit existing multimodal encoders with MultiFair's training procedure and improve both average accuracy and subgroup equity without changing the inference-time model.
  • The method needs only per-group labels during training (which are already collected in most clinical studies) and a differentiable surrogate for AUC, so it is applicable beyond glaucoma, e.g., to diabetic retinopathy, chest X-ray with reports, or any paired imaging-plus-text diagnosis.
  • The threshold τ makes fairness modulation conditional; when groups are already balanced, training reduces to standard balanced multimodal learning, so the added fairness machinery does not slow training unnecessarily.
  • The reported gains in ES-AUC show that fairness metrics improve in the equity-scaled sense even on datasets where the overall AUC of the privileged group slightly drops, implying a better performance-fairness frontier than existing trade-off methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's scalar f_batch multiplies the entire encoder gradient, so it cannot by itself direct gradient signal toward one demographic subgroup; any group-specific emphasis must come from the gradients of the fairness loss F_G, a pathway the paper never derives. A reader should treat 'emphasize updating the respective group' as an interpretation rather than a proven mechanism.
  • The manuscript's abstract describes multiclass and missing-modality settings, while the full text reports only two-modality binary experiments with complete paired data; the claimed generality for those settings is not demonstrated in this paper.
  • A clean ablation—keeping f_batch while removing F_G, or keeping F_G while setting f_batch=1—would separate which component actually moves group AUCs. The paper's ablation only toggles whole modules.
  • The convergence guarantee relies on a strictly positive lower bound β_min for the combined modulation factor; in heavily imbalanced batches this bound can be very small, making the descent term −αβ_min λ_j||∇f_j||^2 negligible in exactly the high-disparity regime the method targets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes MultiFair, a training-time gradient modulation method for multimodal medical classification that aims to jointly mitigate modality imbalance and demographic unfairness. The method combines a classifier-guided modality balancing factor B_i, a gradient-direction alignment loss L_gm, and a fairness-aware factor f_batch_i computed from EMA-based surrogate AUC gaps between demographic groups; the encoder update in Eq. 15 multiplies the gradient by B_i f_batch_i, and the total loss adds a fairness gap F_G. Experiments on the FairVision and FairCLIP datasets report AUC, ES-AUC, subgroup AUCs, DPD, and DeOdds, claiming consistent superiority over unimodal, fairness-aware, and balanced multimodal baselines, with an ablation study and parameter sensitivity analysis.

Significance. If the reported improvements are reproducible, MultiFair would offer a simple training-time method that improves both predictive accuracy and equity in multimodal medical classification, and the use of EMA-based surrogate AUCs for fairness-sensitive gradient modulation is an interesting idea. The paper evaluates on relevant benchmarks and compares with several strong baselines. However, the manuscript contains no machine-checked proofs or reproducible code, the theoretical convergence proof is only a sketch with unquantified assumptions, and the empirical claims are based on single point estimates without uncertainty quantification. Most importantly, the central mechanistic claim of group-level gradient modulation is not supported by the equations as written. These issues make the current version unsuitable for acceptance, but the empirical direction is defensible and the paper could be revised.

major comments (4)
  1. [§IV-C1, IV-D (Eqs. 11–15)] Eq. 11 defines a per-group factor F_i^(g), but Eq. 12 aggregates it into a single scalar f_batch_i = Σ_g p_g F_i^(g) for each modality i, and Eq. 15 multiplies the entire encoder gradient by this scalar. A scalar shared by all samples of a modality cannot "emphasize updating the respective group" as stated in §IV-C1. To achieve group-specific updates one would need sample-level or per-group-loss scaling, e.g. Σ_g F_i^(g) ∇ L_g, not f_batch_i · Σ_g ∇ L_g. The only group-dependent term in the loss is F_G (Eq. 13), but no group-specific gradient is derived for it. As written, the "group level" of the dual-level modulation reduces to a scalar reweighting of each modality plus a scalar fairness penalty; the claimed mechanism is therefore not established.
  2. [§IV-D, Theorem and Proof (Eqs. 16–24)] The convergence proof is not valid as written. Eqs. 18–24 replace the iteration-dependent β(t) by β_max in the quadratic term and by β_min in the descent term; for monotonic decrease one needs αβ_min(λ_j‖∇f_j‖² − C) to dominate α²β_max²L²/2·‖∇L_total‖², but the authors only assume α < 2/(L′β_max), which does not control C or the squared gradient norm. The "sufficiently large λ_j" clause is not quantitative, and the cross-term bound C is never stated. Moreover, β_min > 0 is asserted but not guaranteed: B_i in Eq. 5 can be negative when AUC changes are negative, and F_i^(g) in Eq. 11 can be negative when a group's EMA AUC exceeds the average by more than τ/δ. The theorem does not establish convergence of f_j or ∇f_j → 0.
  3. [Tables I–II and §V-D] The central empirical claim is supported only by point estimates, apparently from a single training run, with no standard deviations, confidence intervals, or significance tests for AUC, ES-AUC, DPD, or DeOdds. This is particularly important because several gains over the strongest baseline (CGGM) are small — e.g. Table I race: AUC 86.40 vs 85.85 and ES-AUC 82.02 vs 80.85 — and the fairness metrics are not consistently better: on FairVision race, MultiFair has DPD 17.28 and DeOdds 16.97 vs CGGM's 12.48 and 7.50. Without uncertainty quantification, the claim that MultiFair "consistently outperforms" these baselines is not established.
  4. [Algorithm 1 vs. §IV-D proof] Algorithm 1 switches between Eq. 6 and Eq. 15 depending on whether ΔAUC_F ≥ τ, so the actual update is piecewise: fairness modulation is applied only when the threshold is exceeded. The theoretical analysis, however, assumes a single update rule with β(t) = αB_i f_batch_i at every iteration. The proof does not account for the threshold gating, the alternation between Eq. 6 and Eq. 15, or the fact that f_batch_i is absent in some iterations. Thus the convergence argument does not cover the algorithm that is implemented and evaluated.
minor comments (5)
  1. [Abstract / §VI] The abstract mentions multiclass classification and missing-modality settings, but the experiments are binary classification with complete paired modalities, and the Discussion states that handling missing/unpaired modalities is future work. Please align the abstract and claims with the actual experiments.
  2. [Eq. 7] The modulation loss L_gm is defined using current gradients and B_i, but the paper does not specify how this term is differentiated when added to L_task. Since Eq. 8 treats L_gm as a backpropagated loss, the gradient computation should be made explicit or the term should be described as a non-differentiated penalty.
  3. [§V-A and References] The paper refers to "FairCLIP" both as a dataset (Sec. V-A) and as a baseline method (Sec. V-B). Reference [13] is the FairCLIP method paper; the dataset name should be clarified to avoid confusion.
  4. [Throughout] There are several typos and inconsistent notations: "fariness" in Eq. 14 context, "upadate" in Algorithm 1, "whcih" in Sec. V-C, and "Fig. V-E" in Sec. V-E should be "Fig. 3".
  5. [Eq. 5] The denominator Σ_k ΔA_k can be zero or very small when modal AUCs change little; the paper does not discuss numerical safeguards. A small epsilon or a fallback rule would improve robustness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fairness and modality modulation terms are training-time feedback signals evaluated on held-out AUC/ES-AUC, so the main results are not forced by construction or by self-citation.

full rationale

MultiFair's derivation chain is self-contained. The training objective (Eq. 14) combines a task loss, a gradient-alignment loss L_gm, and a differentiable AUC-based fairness gap F_G; the gradient update (Eq. 15) is a standard loss-modulated SGD step. Reported AUC and ES-AUC are computed on held-out data, so the empirical gains are not identical by construction to the surrogate fairness gap F_G that is optimized during training. The self-citations to FairVision, FairCLIP, and the Harvard Glaucoma Fairness metric are used as datasets, baselines, and evaluation metrics, not as load-bearing justification for the method's correctness; these are external benchmarks and standard practice. The theoretical convergence argument is a generic smoothness/descent proof; even if it contains notational or technical weaknesses, those are correctness concerns rather than circularity. No fitted parameter is renamed as a prediction, and no equation reduces to its own input by definition.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method's predictions are not forced by definition, but it depends on several hand-set hyperparameters and on unverified assumptions about surrogate AUC and group-level gradient steering. There are no new entities beyond the modulation factors, which are algorithmic constructs.

free parameters (6)
  • ρ = 1.2
    Modality balancing strength in Eq. 5; chosen by hand; sensitivity shown in Fig. 5b.
  • λ_gm = 0.15
    Weight for gradient-modulation loss in Eq. 8; chosen by hand.
  • τ = 0.04/0.02 for FairVision gender/race; 0.07 for FairCLIP
    Fairness trigger threshold; tuned separately per attribute and dataset.
  • δ = 0.3/0.6/0.5
    Fairness modulation strength in Eq. 11; tuned per dataset/attribute.
  • λ_f = 0.5/0.6/0.5
    Fairness penalty weight in Eq. 14; tuned per dataset/attribute.
  • s (EMA smoothing) = not specified
    EMA smoothing factor in Eq. 9; value never reported.
assumptions (4)
  • domain assumption The differentiable margin-based surrogate AUC is a faithful proxy for ranking AUC and can be tracked per group via EMA.
    Eqs. 9/11 rely on this to make fairness decisions; no derivation or validation is provided.
  • standard math Losses L_task, L_gm, F_G are L-smooth with bounded gradients.
    Used in the convergence theorem; standard but unverified for the actual neural network.
  • domain assumption Per-batch group proportions p_g=N_g/N are sufficient to estimate group-specific modulation even for small groups (e.g., 8.5% Asian).
    Eq. 12 averages factors by batch proportions; no minimum group-count analysis is given.
  • ad hoc to paper Fairness modulation can be switched off by threshold τ without harming convergence.
    Algorithm 1 line 9 selectively applies the fairness loss; the proof does not model this switching.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulation." pith.science (2026). https://pith.science/paper/U44B4QRI

@misc{pith2026251007328,
  author       = {Pith},
  title        = {Pith review of: MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U44B4QRI}},
  note         = {Machine review of arXiv:2510.07328}
}
read the original abstract

Medical decision systems increasingly rely on data from multiple sources to ensure reliable and unbiased diagnosis. However, existing multimodal learning models fail to achieve this goal because they often overlook two critical challenges. First, various data modalities may learn unevenly, thereby converging to a model biased towards certain modalities. Second, the model may emphasize learning on certain demographic groups causing unfair performances. The two aspects can influence each other, as different data modalities may favor respective groups during optimization, leading to both imbalanced and unfair multimodal learning. This paper proposes a novel approach called MultiFair for multimodal medical classification, which addresses these challenges with a dual-level gradient modulation process. MultiFair dynamically modulates training gradients regarding the optimization direction and magnitude at both data modality and group levels. We evaluate MultiFair on three real-world medical classification datasets with diverse demographic attributes,including multiclass classification and missing-modality settings. Experimental results demonstrate its effectiveness.

Figures

Figures reproduced from arXiv: 2510.07328 by the authors.

Figure 1
Figure 1. The differences between existing and proposed multimodal learning paradigms. SLO: scanning laser ophthalmoscopy. OCT: opti￾cal coherence tomography. diseases [2]. For example, it is common that an ophthalmol￾ogist combine information from multiple observations like retinal fundus photos, optical coherence tomography (OCT) scans, and clinical notes for evaluating glaucoma patients ( [PITH_FULL_IMAGE:figures/full_fig… view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Ablation study results. The performance of MultiFairG with fairness only, MultiFairM with modality only, and MultiFair with both modulation. Fig.3a represents the performance for the gender subgroup (Male and Female) with and without fairness modulation. Fig. 3b shows the corresponding performance for various racial subgroups. ities, unimodal models such as 3D OCT and text-only models achieve reasonable performance … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The influence of different fairness parameters on performance, measured in terms of AUC, ES-AUC, Group AUCs (Male and Female). Fig. (a) represents the variation in AUCs across different fairness thresholds (τ), while Fig. (b) shows the effect of the fairness penalty λf…
Figure 5
Figure 5. Figure 5: The impact of the modality balancing factors. Fig. (a) λgm and Fig. (b) ρ on the performance in terms of AUC and ES-AUC. except for the Male subgroup, which exhibits greater sensitiv￾ity as the privileged group. Fig. 4c further indicates that the model is highly sensit…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 7 linked inside Pith

  1. [1]

    Advancing clinical practice: The potential of multimodal technology in modern medicine,

    Y . Artsi, V . Sorin, B. S. Glicksberg, G. N. Nadkarni, and E. Klang, “Advancing clinical practice: The potential of multimodal technology in modern medicine,”Journal of Clinical Medicine, vol. 13, no. 20, p. 6246, 2024

    2024

  2. [2]

    Multimodal machine learning in precision health: A scoping review,

    A. Kline, H. Wang, Y . Li, S. Dennis, M. Hutch, Z. Xu, F. Wang, F. Cheng, and Y . Luo, “Multimodal machine learning in precision health: A scoping review,”NPJ digital medicine, vol. 5, no. 1, p. 171, 2022

    2022

  3. [3]

    Detection and prognostic significance of optic disc hemorrhages during the ocular hypertension treatment study,

    D. L. Budenz, D. R. Anderson, W. J. Feuer, J. A. Beiser, J. Schiffman, R. K. Parrish II, J. R. Piltz-Seymour, M. O. Gordon, M. A. Kass, O. H. T. S. Groupet al., “Detection and prognostic significance of optic disc hemorrhages during the ocular hypertension treatment study,” Ophthalmology, vol. 113, no. 12, pp. 2137–2143, 2006

    2006

  4. [4]

    Detection of glaucoma and its association with diabetic retinopathy in a diabetic retinopathy screening program,

    R. A. Gangwani, S. M. McGhee, J. S. Lai, C. K. Chan, and D. Wong, “Detection of glaucoma and its association with diabetic retinopathy in a diabetic retinopathy screening program,”Journal of glaucoma, vol. 25, no. 1, pp. 101–105, 2016

    2016

  5. [5]

    A survey of multimodal learning: Methods, applications, and future,

    Y . Yuan, Z. Li, and B. Zhao, “A survey of multimodal learning: Methods, applications, and future,”ACM Computing Surveys, vol. 57, no. 7, pp. 1–34, 2025

    2025

  6. [6]

    Multimodal biomedical ai,

    J. N. Acosta, G. J. Falcone, P. Rajpurkar, and E. J. Topol, “Multimodal biomedical ai,”Nature medicine, vol. 28, no. 9, pp. 1773–1784, 2022

    2022

  7. [7]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  8. [8]

    Alignmamba: Enhancing multimodal mamba with local and global cross-modal align- ment,

    Y . Li, Y . Xing, X. Lan, X. Li, H. Chen, and D. Jiang, “Alignmamba: Enhancing multimodal mamba with local and global cross-modal align- ment,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 24 774–24 784

    2025

Show all 51 references
  1. [9]

    Deep multimodal data fusion,

    F. Zhao, C. Zhang, and B. Geng, “Deep multimodal data fusion,”ACM computing surveys, vol. 56, no. 9, pp. 1–36, 2024. 10

    2024

  2. [10]

    Multimodal alignment and fusion: A survey,

    S. Li and H. Tang, “Multimodal alignment and fusion: A survey,”arXiv preprint arXiv:2411.17040, 2024

    2024

  3. [11]

    On-the-fly modulation for balanced multimodal learning,

    Y . Wei, D. Hu, H. Du, and J.-R. Wen, “On-the-fly modulation for balanced multimodal learning,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  4. [12]

    Classifier-guided gradient modulation for enhanced multimodal learning,

    Z. Guo, T. Jin, J. Chen, and Z. Zhao, “Classifier-guided gradient modulation for enhanced multimodal learning,”Advances in Neural Information Processing Systems, vol. 37, pp. 133 328–133 344, 2024

    2024

  5. [13]

    Fairclip: Harnessing fairness in vision-language learning,

    Y . Luo, M. Shi, M. O. Khan, M. M. Afzal, H. Huang, S. Yuan, Y . Tian, L. Song, A. Kouhana, T. Elzeet al., “Fairclip: Harnessing fairness in vision-language learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 12 289–12 301

    2024

  6. [14]

    Fairmedfm: fairness benchmarking for medical imaging foundation models,

    R. Jin, Z. Xu, Y . Zhong, Q. Yao, D. QI, S. K. Zhou, and X. Li, “Fairmedfm: fairness benchmarking for medical imaging foundation models,”Advances in Neural Information Processing Systems, vol. 37, pp. 111 318–111 357, 2024

    2024

  7. [15]

    Explainable deep learning methods in medical image classification: A survey,

    C. Patr ´ıcio, J. C. Neves, and L. F. Teixeira, “Explainable deep learning methods in medical image classification: A survey,”ACM Computing Surveys, vol. 56, no. 4, pp. 1–41, 2023

    2023

  8. [16]

    Diagnosing and re-learning for balanced multimodal learning,

    Y . Wei, S. Li, R. Feng, and D. Hu, “Diagnosing and re-learning for balanced multimodal learning,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 71–86

    2024

  9. [17]

    On the benefits of early fusion in multimodal representation learning,

    G. Barnum, S. Talukder, and Y . Yue, “On the benefits of early fusion in multimodal representation learning,”arXiv preprint arXiv:2011.07191, 2020

    2011 arXiv

  10. [18]

    Mmtm: Multimodal transfer module for cnn fusion,

    H. R. V . Joze, A. Shaban, M. L. Iuzzolino, and K. Koishida, “Mmtm: Multimodal transfer module for cnn fusion,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 13 289–13 299

    2020

  11. [19]

    Multimodal transformer for unaligned multimodal language sequences,

    Y .-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” inProceedings of the conference. Association for computational linguistics. Meeting, vol. 2019, 2019, p. 6558

    2019

  12. [20]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” inInternational conference on machine learning. PMLR, 2021, pp. 4904–4916

    2021

  13. [21]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,”Advances in neural information processing systems, vol. 32, 2019

    2019

  14. [22]

    Uniter: Universal image-text representation learning,

    Y .-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y . Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in European conference on computer vision. Springer, 2020, pp. 104– 120

    2020

  15. [23]

    Visualbert: A simple and performant baseline for vision and language,

    L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,”arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  16. [24]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    2023

  17. [25]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynoldset al., “Flamingo: a visual language model for few-shot learning,”Advances in neural information processing systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  18. [26]

    Vlmt: Vision-language multimodal transformer for multimodal multi- hop question answering,

    Q. Z. Lim, C. P. Lee, K. M. Lim, and K. S. M. Anbananthen, “Vlmt: Vision-language multimodal transformer for multimodal multi- hop question answering,”arXiv preprint arXiv:2504.08269, 2025

    2025 arXiv

  19. [27]

    Crossvit: Cross-attention multi- scale vision transformer for image classification,

    C.-F. Chen, Q. Fan, and R. Panda, “Crossvit: Cross-attention multi- scale vision transformer for image classification,”Proceedings of the International Conference on Computer Vision (ICCV), 2021

    2021

  20. [28]

    Multivit: Multimodal vi- sion transformer for schizophrenia prediction using structural mri and functional network connectivity data,

    Y . Bi, A. Abrol, Z. Fu, and V . Calhoun, “Multivit: Multimodal vi- sion transformer for schizophrenia prediction using structural mri and functional network connectivity data,” in2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), 2023, pp. 1–5

    2023

  21. [29]

    What makes training multi- modal classification networks hard?

    W. Wang, D. Tran, and M. Feiszli, “What makes training multi- modal classification networks hard?” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  22. [30]

    Balanced multimodal learning via on-the-fly gradient modulation,

    X. Peng, Y . Wei, A. Deng, D. Wang, and D. Hu, “Balanced multimodal learning via on-the-fly gradient modulation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 8238–8247

    2022

  23. [31]

    Boosting multi-modal model performance with adaptive gradient modulation,

    H. Li, X. Li, P. Hu, Y . Lei, C. Li, and Y . Zhou, “Boosting multi-modal model performance with adaptive gradient modulation,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 22 214–22 224

    2023

  24. [32]

    Mind the gap: Alleviating local imbalance for unsupervised cross-modality medical image segmentation,

    Z. Su, K. Yao, X. Yang, Q. Wang, Y . Yan, J. Sun, and K. Huang, “Mind the gap: Alleviating local imbalance for unsupervised cross-modality medical image segmentation,”IEEE Journal of Biomedical and Health Informatics, vol. 27, no. 7, pp. 3396–3407, 2023

    2023

  25. [33]

    Class balancing diversity multi- modal ensemble for alzheimer’s disease diagnosis and early detection,

    A. Francesconi, L. di Biase, D. Cappetta, F. Rebecchi, P. Soda, R. Sicilia, V . Guarrasi, A. D. N. Initiativeet al., “Class balancing diversity multi- modal ensemble for alzheimer’s disease diagnosis and early detection,” Computerized Medical Imaging and Graphics, vol. 123, p....

    2025

  26. [34]

    Fairvision: Equitable deep learning for eye disease screening via fair identity scaling,

    Y . Luo, M. O. Khan, Y . Tian, M. Shi, Z. Dou, T. Elze, Y . Fang, and M. Wang, “Fairvision: Equitable deep learning for eye disease screening via fair identity scaling,” 2024. [Online]. Available: https://arxiv.org/abs/2310.02492

    2024 arXiv

  27. [35]

    Fairness-aware multimodal learning in automatic video interview assessment,

    C. Kim, J. Choi, J. Yoon, D. Yoo, and W. Lee, “Fairness-aware multimodal learning in automatic video interview assessment,”IEEE Access, vol. 11, pp. 122 677–122 693, 2023

    2023

  28. [36]

    Fairrefuse: Referee-guided fusion for multimodal causal fairness in depression detection,

    J. Cheong, S. Kalkan, and H. Gunes, “Fairrefuse: Referee-guided fusion for multimodal causal fairness in depression detection,” inInternational Joint Conference on Artificial Intelligence (IJCAI), 2024

    2024

  29. [37]

    Fmbench: Benchmarking fairness in multimodal large language models on medical tasks,

    P. Wu, C. Liu, C. Chen, J. Li, C. I. Bercea, and R. Arcucci, “Fmbench: Benchmarking fairness in multimodal large language models on medical tasks,”arXiv preprint arXiv:2410.01089, 2024

    2024 arXiv

  30. [38]

    Fairehr-clp: Towards fairness-aware clinical predictions with con- trastive learning in multimodal electronic health records,

    Y . Wang, M. Pillai, Y . Zhao, C. Curtin, and T. Hernandez-Boussard, “Fairehr-clp: Towards fairness-aware clinical predictions with con- trastive learning in multimodal electronic health records,”arXiv preprint arXiv:2402.00955, 2024

    2024 arXiv

  31. [39]

    Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification,

    Z. Yuan, Y . Yan, M. Sonka, and T. Yang, “Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3040–3049

    2021

  32. [40]

    Acceleration of stochastic approxima- tion by averaging,

    B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approxima- tion by averaging,”SIAM Journal on Control and Optimization, vol. 30, no. 4, pp. 838–855, 1992

    1992

  33. [41]

    Nesterovet al.,Lectures on convex optimization

    Y . Nesterovet al.,Lectures on convex optimization. Springer, 2018, vol. 137

    2018

  34. [42]

    Goodfellow, Y

    I. Goodfellow, Y . Bengio, and A. Courville,Deep Learning. MIT Press, 2016. [Online]. Available: https://www.deeplearningbook.org

    2016

  35. [43]

    Efficientnet: Rethinking model scaling for convolutional neural networks,

    M. Tan and Q. V . Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” inProceedings of the 36th International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, pp. 6105–6114

    2019

  36. [44]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778

    2016

  37. [45]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”International Conference on Learning Representations (ICLR), 2015, arXiv:1409.1556

    2015 arXiv

  38. [46]

    An image is worth 16x16 words: Trans- formers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Trans- formers for image recognition at scale,” inInternational Conference on Learnin...

    2021

  39. [47]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inPro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolog...

    2019

  40. [48]

    On-the-fly modulation for balanced multimodal learning,

    Y . Wei, D. Hu, H. Du, and J.-R. Wen, “On-the-fly modulation for balanced multimodal learning,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, early access as of September 2024

    2025

  41. [49]

    Harvard glaucoma fairness: A retinal nerve disease dataset for fairness learning and fair identity normalization,

    Y . Luo, Y . Tian, M. Shi, L. R. Pasquale, L. Q. Shen, N. Zebardast, T. Elze, and M. Wang, “Harvard glaucoma fairness: A retinal nerve disease dataset for fairness learning and fair identity normalization,” IEEE Transactions on Medical Imaging, pp. 1–1, 2024

    2024

  42. [50]

    A reductions approach to fair classification,

    A. Agarwal, A. Beygelzimer, M. Dud ´ık, J. Langford, and H. Wallach, “A reductions approach to fair classification,” inProceedings of the 35th International Conference on Machine Learning (ICML). PMLR, 2018, pp. 60–69

    2018

  43. [51]

    Fair regression: Quantitative definitions and reduction-based algorithms,

    A. Agarwal, M. Dud ´ık, and Z. S. Wu, “Fair regression: Quantitative definitions and reduction-based algorithms,” inProceedings of the 36th International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, pp. 120–129

    2019

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.