Pith. sign in

REVIEW 3 major objections 6 minor 126 references

Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multimodal federated learning is best organized by the underlying federated-learning paradigm, with horizontal, vertical, and hybrid settings each posing its own dominant challenge.

desk verdict Useful paradigm-based survey of MFL; read it for the overview, but the hybrid FL branch is thinner and more self-referential than the 'taxonomy of existing work' claim admits. read the letter →

arxiv 2505.21792 v1 pith:QCBUKHQ2 submitted 2025-05-27 cs.LG cs.AI

Yuanzhe Peng , Jieming Bian , Lei Wang , Yin Huang , Jie Xu This is my paper
classification cs.LGcs.AI
keywords MultimodalFederatedLearningHorizontalSample-spacePartitionVerticalFeature-spaceHybridPartitioningModalityHeterogeneityPrivacyandSecurityComputationalEfficiencyCommunication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal federated learning (MFL) combines multiple data modalities with distributed, privacy-preserving training, and this survey claims that the field is best organized by the way data are split across clients. It proposes a taxonomy with three branches — horizontal FL (same features, different samples), vertical FL (same samples, different features), and hybrid FL (both split) — and for each branch it gives a problem formulation, representative algorithms, and the challenge that multimodality makes most acute. The claimed dominant challenges are modality heterogeneity in horizontal FL, privacy leakage in vertical FL, and efficiency in hybrid FL. A reader should care because this framing turns a scattered literature into a structured map and predicts that solutions will not transfer across paradigms.

What carries the argument

The carrying object is the taxonomy itself, built on the three classical FL data-partitioning schemes: horizontal, vertical, and hybrid. Each branch is analyzed along three fixed dimensions — problem formulation, training algorithm, and key challenge — so that heterogeneous methods can be compared within a consistent structure. The taxonomy is what does the argumentative work: it assigns each surveyed method to a paradigm, attaches a dominant challenge to that paradigm, and then uses the challenge to explain why standard FL tools (such as FedAvg's shared-feature-space aggregation) fail in multimodal settings.

What would settle it

A concrete falsifier would be to take any proposed hybrid-FL method and reimplement it as a nested or alternating sequence of a horizontal-FL method and a vertical-FL method, then show the composed system matches the hybrid method in accuracy, privacy, and communication cost; success would show hybrid FL is not a separate paradigm. A complementary falsifier would be a hybrid-FL system whose measured bottleneck is privacy leakage or modality heterogeneity rather than efficiency, contradicting the claim that efficiency is the most fundamental concern in that paradigm.

Watch

Extended reading notes

Core claim

The central claim is that existing MFL research falls naturally into three federated-learning paradigms, and that this is the first taxonomy to organize the field that way. In horizontal FL, clients share a feature space but own different samples, and the paper identifies modality heterogeneity — clients holding different subsets of modalities — as the most critical obstacle. In vertical FL, clients own different features of the same samples, and the exchange of intermediate embeddings makes privacy leakage the most critical challenge. In hybrid FL, both the sample space and the feature space are partitioned, and efficiency becomes the most fundamental practical concern. The survey backs these claims with problem formulations, training algorithms, and catalogs of representative methods and attacks, and it concludes that each paradigm's challenges are qualitatively different from those in unimodal FL or centralized multimodal learning.

Load-bearing premise

The taxonomy's third branch assumes that hybrid FL — simultaneous partitioning of the sample space and feature space — is a genuinely distinct paradigm and not just a composition of horizontal and vertical FL; the paper itself concedes that the setting reduces to the HFL or VFL cases when only one dimension is partitioned.

Editorial extensions

If this is right

  • If the taxonomy is correct, a researcher encountering a new MFL method can classify it by data-partition geometry and immediately know which challenge to design for.
  • In multimodal HFL, the taxonomy implies that aggregation rules assuming a common feature space will fail when clients hold different modality subsets, motivating modality-aware aggregation.
  • In multimodal VFL, the survey's challenge analysis implies that privacy defenses for embeddings and gradients, not efficiency measures, should be the primary design target.
  • In hybrid FL, the taxonomy implies that efficiency — computational and communication overhead from dual-axis partitioning — is the bottleneck that needs resource-management and compression techniques.
  • The compiled dataset and application catalog gives practitioners a concrete starting point for benchmarking MFL methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the taxonomy is to vary the sample-space/feature-space partition ratio in hybrid FL and measure when the efficiency bottleneck displaces privacy or heterogeneity as the dominant concern.
  • The taxonomy's logic extends beyond multimodality: any client-side feature-space heterogeneity (e.g., differing sensor suites, label spaces, or missing features) could be analyzed along the same paradigm axes.
  • The hybrid-FL branch currently rests on a narrow base: the paper's running example and efficiency discussion are drawn largely from the authors' own prior work, so an independent implementation of hybrid FL with different models and datasets would test whether the paradigm stands on its own.
  • If the per-paradigm dominant challenges hold, they suggest a meta-design rule: choose aggregation, privacy, and compression strategies from the data-partition geometry rather than from the application domain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This survey proposes a taxonomy of multimodal federated learning (MFL) organized by three federated learning (FL) paradigms: horizontal FL (HFL), vertical FL (VFL), and hybrid FL. For each paradigm, the paper provides a problem formulation, reviews representative algorithms, identifies a key challenge, and discusses applications and datasets. The main contributions claimed are: (i) a novel taxonomy that is the first to systematically organize existing MFL research by the underlying FL paradigm; (ii) problem formulations, algorithm reviews, and per-paradigm challenge analyses; (iii) a compilation of datasets and applications; and (iv) a discussion of open challenges and future directions. The paper's central narrative is that multimodal data introduces distinct challenges in each FL paradigm, with modality heterogeneity being the most critical challenge in HFL, privacy leakage in VFL, and efficiency in hybrid FL.

Significance. If the taxonomy and the per-paradigm challenge claims are accepted, the paper would provide a useful organizing framework for a growing literature at the intersection of multimodal learning and federated learning. The survey's strengths include broad coverage of recent MFL works, a systematic presentation of HFL and VFL problem formulations, a reasonably detailed catalog of privacy attacks in VFL, and a practical list of datasets and applications. These features make the manuscript a potentially valuable reference for researchers entering the field. However, the significance is currently tempered by two load-bearing issues: the hybrid FL branch is defined in a way that reduces to a composition of HFL and VFL and is operationalized almost entirely through the authors' prior work, and the HFL problem formulation excludes the very modality heterogeneity that the paper later identifies as the central HFL challenge. Both issues affect the validity of the central claim that the paper offers a taxonomy grounded in three core FL paradigms.

major comments (3)
  1. [§5, §2] The hybrid FL branch is not convincingly established as a distinct third paradigm. Section 5 defines hybrid FL as simultaneous partitioning of both the sample space and the feature space, but immediately concedes that the term is non-standard and that 'when reduced to a single-dimensional partition, this setting corresponds to either the HFL or VFL cases.' This makes hybrid FL a composition of the other two paradigms rather than a primitive one. Moreover, the only described training workflow in §5.2 is 'the hybrid FL training workflow introduced in [77]', and §5.3 repeatedly points to [77, 82]; all of these are papers by the same research group. No independent body of literature is classified under the hybrid branch. Since the paper's central claim in §1 is that it presents a taxonomy of 'existing MFL research' grounded in three core FL paradigms, the hybrid branch as presented is better described as a proposed setting rather than a surveyed category. The authors should either reframe the contribution as a two-paradigm survey plus a research proposal for hybrid FL, or provide independent examples demonstrating that hybrid FL with simultaneous sample-space and feature-space partitioning is an established paradigm with a substantial literature.
  2. [§3.1, §3.4] There is a direct internal inconsistency between the problem formulation and the stated key challenge of HFL. Section 3.1 formalizes multimodal HFL under the assumption that 'all clients share the same set of modalities' and that 'each sample contains the same types of features.' Section 3.4 then declares modality heterogeneity, i.e., clients holding different subsets of modalities, to be 'the most critical challenge in multimodal HFL.' These two statements are mutually exclusive: the formal setting in Eq. (1) and the local dataset definition D_m = {(x_i_m, y_i_m)} exclude the modality-heterogeneous scenario. The authors should either generalize the problem formulation to accommodate heterogeneous modality availability, or explicitly present §3.1 as the consistent-modality special case and formalize the modality-heterogeneous setting separately before discussing challenges.
  3. [§3.4, §4.3, §5.3] The paper's per-paradigm 'most critical challenge' claims are asserted without comparative support. Section 3.4 states that modality heterogeneity is 'the most critical challenge' in HFL, §4.3 states that privacy leakage is 'one of the most critical challenges' in VFL, and §5.3 states that efficiency 'stands out as the most fundamental and practical concern' in hybrid FL. The survey does not provide any structured comparison of challenge importance, nor does it present evidence from the reviewed works that would justify these prioritizations. For instance, the VFL discussion in §4.3 catalogs a wide range of privacy attacks but does not compare their severity with, say, communication overhead or feature imbalance, which are mentioned only in passing. The authors should either (i) add a comparative analysis that justifies why one challenge is more critical than others within each paradigm, or (ii) soften these claims to say that these are important challenges rather than the most critical ones.
minor comments (6)
  1. [§4.1] The definition 'h0(θ0; xi) := θ0' is confusing: the server head is treated both as a parameter and as an embedding function. Consider writing the objective in Eq. (1) with an explicit server-head function applied to the concatenation of client embeddings, e.g., ℓ(h0(θ0; [h1(θ1; x_i^1), ..., hK(θK; x_i^K)]), y_i).
  2. [§5.1] The phrase 'bi-orthogonally partitioned' is used without definition. Please clarify what 'bi-orthogonal' means in this context, or replace it with a more standard term such as 'both horizontally and vertically partitioned.'
  3. [§5.2] The sentence 'Every RQ updates, the global server aggregates the local models' is ambiguous. Specify whether R refers to the number of global rounds and Q to the number of local updates per round, and define the notation before using it.
  4. [§6.2, Table 3] The dataset 'SEW A' is listed in Table 3 without a citation. Either provide a reference for this dataset or remove it from the table.
  5. [§3.3, §6.1] The abbreviation 'FedMEKT' is used for two different works: reference [36] in Table 1 and reference [89] in Section 6.1. Please disambiguate the names, for example by using the full author names or a version number, to avoid confusion.
  6. [§6.1] In the discussion of embodied intelligence, the paper cites [92] as 'emerging' research, but the reference is a Nature Communications paper on embodied intelligence via learning and evolution. Please verify that this citation is appropriate for the claim being made about federated multimodal robot learning.

Circularity Check

1 steps flagged · score 4.0 of 10

The central HFL/VFL axes are externally grounded, but the survey's distinctive third branch (hybrid FL) is defined as a composition of the other two and is populated almost entirely by the authors' own prior work ([77], [82]), making the 'first taxonomy of existing MFL research' claim partially self-citational.

  1. self citation load bearing [Section 5, 'Multimodal Hybrid Federated Learning' (definition in the section opening, workflow example in §5.2, efficiency challenge in §5.3)]
    "Although some studies use the term "hybrid FL" to describe scenarios where both the client and the server hold training data, in this paper, we define hybrid FL as the case where both the sample space and the feature space are partitioned across clients. ... For an example of the hybrid FL training workflow introduced in [77], each silo contains K vertical parties that engage in vertical FL, while M silos communicate horizontally through periodic global aggregation."

    The paper's central claim is to be the first to organize existing MFL research into three core paradigms, with hybrid FL as the novel third branch. But the paper itself concedes that this branch is a composition of the first two ('reduced to a single-dimensional partition, this setting corresponds to either the HFL or VFL cases'), and the only concrete workflow presented under the branch is 'the hybrid FL training workflow introduced in [77]', an earlier paper by the same University of Florida group (Peng, Wu, Bian, Xu). Section 5.3 again anchors efficiency as the defining challenge on '[77, 82]', also the authors' own prior works.

full rationale

Most of the paper's organizational scaffolding is externally grounded: HFL and VFL are standard FL paradigms with citations to Kairouz et al., Liu et al., and many independent MFL papers are classified under those branches (FedMultimodal, MMFed, FedMSplit, MMVFL, etc.). The circularity risk is concentrated in the third branch. Section 5 defines hybrid FL as simultaneous sample/feature partitioning, concedes the term is non-standard ('Although some studies use the term ...'), concedes that a one-dimensional partition reduces to HFL or VFL, then gives [77] as the example workflow and later anchors the efficiency challenge on [77, 82]; both references are the same University of Florida group. The survey's 'first to systematically organize MFL research' claim therefore depends on counting the authors' own proposed setting as an established paradigm; that step is load-bearing self-citation rather than a survey of independent literature. This is not a case of a fitted parameter renamed as prediction, and the HFL/VFL halves retain independent content, so the score is moderate rather than high. The paper's internal inconsistency between §3.1's assumption that 'all clients share the same set of modalities' and §3.4's claim that modality heterogeneity is the most critical HFL challenge is a correctness/consistency issue, not a circularity issue.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted parameters and no new physical or mathematical entities. Its weight-bearing assumptions are organizational: the exhaustiveness of the three paradigms, the shared-modality premise in the HFL model that contradicts the HFL challenge narrative, and the unverified negative claim about prior taxonomies. The hybrid FL branch functions as the paper's invented classification axis, grounded mainly in the authors' own publications, which the circularity score (4) reflects.

assumptions (4)
  • domain assumption The three FL paradigms (HFL, VFL, hybrid) form a complete partition of federated settings, with hybrid as a co-equal third paradigm.
    Invoked throughout §2 and §5. The paper asserts hybrid FL as a major paradigm while conceding the term is used differently elsewhere and that single-dimensional limits reduce it to HFL or VFL.
  • domain assumption The formal HFL model assumes all clients share the same set of modalities.
    §3.1 states 'All clients share the same set of modalities', which is precisely the assumption §3.4 criticizes in prior work. The survey's own model is inconsistent with its headline HFL challenge.
  • ad hoc to paper No prior work has systematically organized MFL through FL paradigms.
    Abstract and §2 assert this negative claim ('To the best of our knowledge, no existing work...') without a documented search protocol or structured comparison with cited surveys such as [7]. The taxonomy's novelty rests on this unverified premise.
  • domain assumption The four-category VFL attack taxonomy (representation-based inference, gradient-based reconstruction, label inference, model stealing) is complete for multimodal VFL.
    §4.3 organizes the cited attack literature into these four categories and asserts privacy leakage as the defining VFL challenge; completeness for the multimodal setting is asserted, not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms." pith.science (2026). https://pith.science/paper/QCBUKHQ2

@misc{pith2026250521792,
  author       = {Pith},
  title        = {Pith review of: Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QCBUKHQ2}},
  note         = {Machine review of arXiv:2505.21792}
}
read the original abstract

Multimodal Federated Learning (MFL) lies at the intersection of two pivotal research areas: leveraging complementary information from multiple modalities to improve downstream inference performance and enabling distributed training to enhance efficiency and preserve privacy. Despite the growing interest in MFL, there is currently no comprehensive taxonomy that organizes MFL through the lens of different Federated Learning (FL) paradigms. This perspective is important because multimodal data introduces distinct challenges across various FL settings. These challenges, including modality heterogeneity, privacy heterogeneity, and communication inefficiency, are fundamentally different from those encountered in traditional unimodal or non-FL scenarios. In this paper, we systematically examine MFL within the context of three major FL paradigms: horizontal FL (HFL), vertical FL (VFL), and hybrid FL. For each paradigm, we present the problem formulation, review representative training algorithms, and highlight the most prominent challenge introduced by multimodal data in distributed settings. We also discuss open challenges and provide insights for future research. By establishing this taxonomy, we aim to uncover the novel challenges posed by multimodal data from the perspective of different FL paradigms and to offer a new lens through which to understand and advance the development of MFL.

Figures

Figures reproduced from arXiv: 2505.21792 by the authors.

Figure 1
Figure 1. MFL draws inspiration from hu￾man multi-sensory collaborative learning. Federated Learning (FL) addresses this need by en￾abling multiple clients to train a shared model collab￾oratively while keeping local data private [8]. FL has demonstrated success in various unimodal tasks such as next-word prediction, image classification, and activ￾ity recognition [9–13]. Nevertheless, directly applying FL to multimodal learn… view at source ↗
Figure 2
Figure 2. (a) HFL addresses horizontally partitioned sample spaces with consistent feature spaces. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Our proposed taxonomy presents MFL from the perspective of different FL paradigms. The [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Multimodal HFL, where each client shares the same multimodal feature space but holds a different sample space. Xiong et al. [15] proposed MMFed, a unified frame￾work that combines co-attention mechanisms with personalization via Model-Agnostic Meta-Learning (MAML) [16]…
Figure 5
Figure 5. Figure 5: The problem of modality heterogeneity in computational pathology. Modality heterogeneity in this context arises from several factors. First, hardware limitations at different hospitals may restrict the use of cer￾tain diagnostic devices, leading to limited data modalit…
Figure 6
Figure 6. Figure 6: Multimodal VFL. Each client trains its local model θk and uploads em￾beddings to the server, which concatenates the embeddings and feeds them into the server head model θ0. The training proceeds in global rounds, where each round r ∈ [R] consists of local updates perfo…
Figure 7
Figure 7. Figure 7: A multimodal system involves problem decomposition across feature and sample spaces. This setup is further complicated by the pres￾ence of diverse data modalities, such as vision, language, audio, and time-series signals. These modalities vary in size, structure, and c…
Figure 9
Figure 9. Figure 9: Multimodal emotion recognition [100] based on IEMOCAP dataset. In MFL settings, clients can represent differ￾ent participants or devices contributing partial modalities (e.g., only audio or only video), en￾abling research on cross-device and cross-view training under i…
Figure 8
Figure 8. Figure 8: Example classes from the Kinetics dataset [ [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 10
Figure 10. Figure 10: Flickr30K dataset examples [105]. In MFL scenarios, it can be used to simulate dis￾tributed content contributors, where each client holds a unique subset of image-caption pairs, enabling federated training of cross-modal align￾ment models. MS COCO [106] is a large-sca…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

126 extracted references · 55 canonical work pages

  1. [77]

    Hybrid federated learning for multimodal iot systems,

    Y . Peng, Y . Wu, J. Bian, and J. Xu, “Hybrid federated learning for multimodal iot systems,” IEEE Internet of Things Journal, 2024

    work page 2024

  2. [1]

    Cross-modal retrieval: a systematic review of methods and future directions,

    T. Wang, F. Li, L. Zhu, J. Li, Z. Zhang, and H. T. Shen, “Cross-modal retrieval: a systematic review of methods and future directions,” Proceedings of the IEEE, 2025

    2025

  3. [2]

    Deep learning approaches on image captioning: A review,

    T. Ghandi, H. Pourreza, and H. Mahyar, “Deep learning approaches on image captioning: A review,”ACM Computing Surveys, vol. 56, no. 3, pp. 1–39, 2023

    2023

  4. [3]

    A comprehensive survey of multi-view video summarization,

    T. Hussain, K. Muhammad, W. Ding, J. Lloret, S. W. Baik, and V . H. C. De Albuquerque, “A comprehensive survey of multi-view video summarization,”Pattern Recognition, vol. 109, p. 107567, 2021

    2021

  5. [4]

    Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data

    P. P. Liang, T. Liu, A. Cai, M. Muszynski, R. Ishii, N. Allen, R. Auerbach, D. Brent, R. Salakhutdinov, and L.-P. Morency, “Learning language and multimodal privacy-preserving markers of mood from mobile data,” arXiv preprint arXiv:2106.13213, 2021

    work page Pith review arXiv 2021

  6. [5]

    Resvit: residual vision transformers for multimodal medical image synthesis,

    O. Dalmaz, M. Yurt, and T. Çukur, “Resvit: residual vision transformers for multimodal medical image synthesis,” IEEE Transactions on Medical Imaging, vol. 41, no. 10, pp. 2598– 2614, 2022

    2022

  7. [6]

    Multimodal federated learning via contrastive representation ensemble,

    Q. Yu, Y . Liu, Y . Wang, K. Xu, and J. Liu, “Multimodal federated learning via contrastive representation ensemble,” arXiv preprint arXiv:2302.08888, 2023

    arXiv 2023

  8. [7]

    Multimodal federated learning in healthcare: a review,

    J. Thrasher, A. Devkota, P. Siwakotai, R. Chivukula, P. Poudel, C. Hu, B. Bhattarai, and P. Gyawali, “Multimodal federated learning in healthcare: a review,” arXiv preprint arXiv:2310.09650, 2023

    arXiv 2023

Show all 126 references
  1. [8]

    Advances and open problems in federated learning,

    P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al., “Advances and open problems in federated learning,” Foundations and trends® in machine learning, vol. 14, no. 1–2, pp. 1–210, 2021

    2021

  2. [9]

    Communication-efficient learning of deep networks from decentralized data,

    H. B. McMahan, E. Moore, D. Ramage, S. Hampson et al., “Communication-efficient learning of deep networks from decentralized data,” Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), 2017

    2017

  3. [10]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine Learning and Systems (MLSys), 2020

    2020

  4. [11]

    Scaffold: Stochastic controlled averaging for federated learning,

    S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in International conference on machine learning. PMLR, 2020, pp. 5132–5143

    2020

  5. [12]

    Fedbn: Federated learning on non-iid features via local batch normalization,

    X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou, “Fedbn: Federated learning on non-iid features via local batch normalization,” arXiv preprint arXiv:2102.07623, 2021

    2021 arXiv

  6. [13]

    Tackling the objective inconsistency problem in heterogeneous federated optimization,

    J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V . Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” Advances in neural information processing systems, vol. 33, pp. 7611–7623, 2020

    2020

  7. [14]

    Multimodal melanoma detection with federated learning,

    B. L. Y . Agbley, J. Li, A. U. Haq, E. K. Bankas, S. Ahmad, I. O. Agyemang, D. Kulevome, W. D. Ndiaye, B. Cobbinah, and S. Latipova, “Multimodal melanoma detection with federated learning,” in 2021 18th international computer conference on wavelet active media technology and i...

    2021

  8. [15]

    A unified framework for multi-modal federated learning,

    B. Xiong, X. Yang, F. Qi, and C. Xu, “A unified framework for multi-modal federated learning,” Neurocomputing, vol. 480, pp. 110–118, 2022

    2022

  9. [16]

    Model-agnostic meta-learning for fast adaptation of deep networks,

    C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in International conference on machine learning. PMLR, 2017, pp. 1126–1135

    2017

  10. [17]

    Towards a multimodal human activity dataset for healthcare,

    M. Hu, M. Luo, M. Huang, W. Meng, B. Xiong, X. Yang, and J. Sang, “Towards a multimodal human activity dataset for healthcare,” Multimedia Systems, vol. 29, no. 1, pp. 1–13, 2023

    2023

  11. [18]

    Medical report generation based on multimodal federated learning,

    J. Chen and R. Pan, “Medical report generation based on multimodal federated learning,” Computerized Medical Imaging and Graphics, vol. 113, p. 102342, 2024. 20

    2024

  12. [19]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

  13. [20]

    Fl-fd: Federated learning-based fall detection with multi- modal data fusion,

    P. Qi, D. Chiaro, and F. Piccialli, “Fl-fd: Federated learning-based fall detection with multi- modal data fusion,” Information fusion, vol. 99, p. 101890, 2023

    2023

  14. [21]

    Fedmultimodal: A benchmark for multimodal federated learning,

    T. Feng, D. Bose, T. Zhang, R. Hebbar, A. Ramakrishna, R. Gupta, M. Zhang, S. Avestimehr, and S. Narayanan, “Fedmultimodal: A benchmark for multimodal federated learning,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 4035–4045

    2023

  15. [22]

    Adaptive federated optimization,

    S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Kone ˇcn`y, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,”arXiv preprint arXiv:2003.00295, 2020

    2003 arXiv

  16. [23]

    Communication- efficient multimodal federated learning: Joint modality and client selection,

    L. Yuan, D.-J. Han, S. Wang, D. Upadhyay, and C. G. Brinton, “Communication- efficient multimodal federated learning: Joint modality and client selection,” arXiv preprint arXiv:2401.16685, 2024

    2024

  17. [24]

    Fedmsplit: Correlation-adaptive federated multi-task learning across multimodal split networks,

    J. Chen and A. Zhang, “Fedmsplit: Correlation-adaptive federated multi-task learning across multimodal split networks,” inProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, 2022, pp. 87–96

    2022

  18. [25]

    On disentanglement of asymmetrical knowledge transfer for modality-task agnostic federated learning,

    ——, “On disentanglement of asymmetrical knowledge transfer for modality-task agnostic federated learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 10, 2024, pp. 11 311–11 319

    2024

  19. [26]

    Autofed: Heterogeneity-aware federated multimodal learning for robust autonomous driving,

    T. Zheng, A. Li, Z. Chen, H. Wang, and J. Luo, “Autofed: Heterogeneity-aware federated multimodal learning for robust autonomous driving,” in Proceedings of the 29th annual international conference on mobile computing and networking, 2023, pp. 1–15

    2023

  20. [27]

    Har- mony: Heterogeneous multi-modal federated learning through disentangled model training,

    X. Ouyang, Z. Xie, H. Fu, S. Cheng, L. Pan, N. Ling, G. Xing, J. Zhou, and J. Huang, “Har- mony: Heterogeneous multi-modal federated learning through disentangled model training,” in Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Se...

    2023

  21. [28]

    Cross-modal federated human activity recognition,

    X. Yang, B. Xiong, Y . Huang, and C. Xu, “Cross-modal federated human activity recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  22. [29]

    Fedfusion: Manifold-driven federated learning for multi- satellite and multi-modality fusion,

    D. Li, W. Xie, Y . Li, and L. Fang, “Fedfusion: Manifold-driven federated learning for multi- satellite and multi-modality fusion,” IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1–13, 2023

    2023

  23. [30]

    Cross-modal federated human activity recognition via modality-agnostic and modality-specific representation learning,

    X. Yang, B. Xiong, Y . Huang, and C. Xu, “Cross-modal federated human activity recognition via modality-agnostic and modality-specific representation learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 3, 2022, pp. 3063–3071

    2022

  24. [31]

    Federated modality-specific encoders and multimodal anchors for personalized brain tumor segmentation,

    Q. Dai, D. Wei, H. Liu, J. Sun, L. Wang, and Y . Zheng, “Federated modality-specific encoders and multimodal anchors for personalized brain tumor segmentation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 2, 2024, pp. 1445–1453

    2024

  25. [32]

    Federated learning for vision-and-language grounding problems,

    F. Liu, X. Wu, S. Ge, W. Fan, and Y . Zou, “Federated learning for vision-and-language grounding problems,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 07, 2020, pp. 11 572–11 579

    2020

  26. [33]

    Towards optimal multi-modal federated learning on non-iid data with hierarchical gradient blending,

    S. Chen and B. Li, “Towards optimal multi-modal federated learning on non-iid data with hierarchical gradient blending,” in IEEE INFOCOM 2022-IEEE conference on computer communications. IEEE, 2022, pp. 1469–1478

    2022

  27. [34]

    Fedclip: Fast generalization and personalization for clip in federated learning,

    W. Lu, X. Hu, J. Wang, and X. Xie, “Fedclip: Fast generalization and personalization for clip in federated learning,” arXiv preprint arXiv:2302.13485, 2023

    2023 arXiv

  28. [35]

    Fedcmr: Federated cross-modal retrieval,

    L. Zong, Q. Xie, J. Zhou, P. Wu, X. Zhang, and B. Xu, “Fedcmr: Federated cross-modal retrieval,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021, pp. 1672–1676. 21

    2021

  29. [36]

    Multimodal federated learning on iot data,

    Y . Zhao, P. Barnaghi, and H. Haddadi, “Multimodal federated learning on iot data,” in2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementa- tion (IoTDI). IEEE, 2022, pp. 43–54

    2022

  30. [37]

    Pfedprompt: Learning personalized prompt for vision-language models in federated learning,

    T. Guo, S. Guo, and J. Wang, “Pfedprompt: Learning personalized prompt for vision-language models in federated learning,” in Proceedings of the ACM Web Conference 2023, 2023, pp. 1364–1374

    2023

  31. [38]

    Multimodal federated learning with missing modality via prototype mask and contrast,

    G. Bao, Q. Zhang, D. Miao, Z. Gong, L. Hu, K. Liu, Y . Liu, and C. Shi, “Multimodal federated learning with missing modality via prototype mask and contrast,” arXiv preprint arXiv:2312.13508, 2023

    2023 arXiv

  32. [39]

    Fedusl: A federated annotation method for driving fatigue detection based on multimodal sensing data,

    S. Yu, Q. Yang, J. Wang, and C. Wu, “Fedusl: A federated annotation method for driving fatigue detection based on multimodal sensing data,” ACM Transactions on Sensor Networks, 2024

    2024

  33. [40]

    Fedsea: Federated learning via selective feature alignment for non-iid multimodal data,

    M. Tan, Y . Feng, L. Chu, J. Shi, R. Xiao, H. Tang, and J. Yu, “Fedsea: Federated learning via selective feature alignment for non-iid multimodal data,”IEEE Transactions on Multimedia, vol. 26, pp. 5807–5822, 2023

    2023

  34. [41]

    Fedmm: Federated multi-modal learning with modality hetero- geneity in computational pathology,

    Y . Peng, J. Bian, and J. Xu, “Fedmm: Federated multi-modal learning with modality hetero- geneity in computational pathology,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 1696–1700

    2024

  35. [42]

    Prioritizing modalities: Flexible importance scheduling in federated multimodal learning,

    J. Bian, L. Wang, and J. Xu, “Prioritizing modalities: Flexible importance scheduling in federated multimodal learning,” arXiv preprint arXiv:2408.06549, 2024

    2024 arXiv

  36. [43]

    A multi-modal vertical federated learning framework based on homomorphic encryption,

    M. Gong, Y . Zhang, Y . Gao, A. K. Qin, Y . Wu, S. Wang, and Y . Zhang, “A multi-modal vertical federated learning framework based on homomorphic encryption,” IEEE Transactions on Information Forensics and Security, vol. 19, pp. 1826–1839, 2023

    2023

  37. [44]

    Toward multimodal vertical federated learning: A traf- fic analysis case study,

    A. P. Sundar, F. Li, X. Zou, and T. Gao, “Toward multimodal vertical federated learning: A traf- fic analysis case study,” in 2024 33rd International Conference on Computer Communications and Networks (ICCCN). IEEE, 2024, pp. 1–9

    2024

  38. [45]

    Cross-modal vertical federated learning for mri reconstruction,

    Y . Yan, H. Wang, Y . Huang, N. He, L. Zhu, Y . Xu, Y . Li, and Y . Zheng, “Cross-modal vertical federated learning for mri reconstruction,”IEEE Journal of Biomedical and Health Informatics, vol. 28, no. 11, pp. 6384–6394, 2024

    2024

  39. [46]

    Vertical Federated Learning: Concepts, Advances, and Challenges,

    Y . Liu, Y . Kang, T. Zou, Y . Pu, Y . He, X. Ye, Y . Ouyang, Y .-Q. Zhang, and Q. Yang, “Vertical Federated Learning: Concepts, Advances, and Challenges,” IEEE Transactions on Knowledge and Data Engineering, vol. 36, pp. 3615–3634, 2024

    2024

  40. [47]

    Compressed-vfl: Communication-efficient learning with vertically partitioned data,

    T. J. Castiglia, A. Das, S. Wang, and S. Patterson, “Compressed-vfl: Communication-efficient learning with vertically partitioned data,” in International Conference on Machine Learning. PMLR, 2022, pp. 2738–2766

    2022

  41. [48]

    Flexible vertical federated learning with heterogeneous parties,

    T. Castiglia, S. Wang, and S. Patterson, “Flexible vertical federated learning with heterogeneous parties,” IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  42. [49]

    Less-vfl: Communication-efficient feature selection for vertical federated learning,

    T. Castiglia, Y . Zhou, S. Wang, S. Kadhe, N. Baracaldo, and S. Patterson, “Less-vfl: Communication-efficient feature selection for vertical federated learning,” in International Conference on Machine Learning. PMLR, 2023, pp. 3757–3781

    2023

  43. [50]

    Fedbcd: A communication- efficient collaborative learning framework for distributed features,

    Y . Liu, X. Zhang, Y . Kang, L. Li, T. Chen, M. Hong, and Q. Yang, “Fedbcd: A communication- efficient collaborative learning framework for distributed features,” IEEE Transactions on Signal Processing, vol. 70, pp. 4277–4290, 2022

    2022

  44. [51]

    Exploiting Shared Representations for Personalized Federated Learning,

    L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploiting Shared Representations for Personalized Federated Learning,” in International conference on machine learning. Virtual Event: PMLR, 2021, pp. 2089–2099

    2021

  45. [52]

    Overlearning reveals sensitive attributes,

    C. Song and V . Shmatikov, “Overlearning reveals sensitive attributes,” arXiv preprint arXiv:1905.11742, 2019. 22

    1905 arXiv

  46. [53]

    Exploiting unintended feature leakage in collaborative learning,

    L. Melis, C. Song, E. De Cristofaro, and V . Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in 2019 IEEE symposium on security and privacy (SP) . IEEE, 2019, pp. 691–706

    2019

  47. [54]

    {ML-Doctor}: Holistic risk assessment of inference attacks against machine learning models,

    Y . Liu, R. Wen, X. He, A. Salem, Z. Zhang, M. Backes, E. De Cristofaro, M. Fritz, and Y . Zhang, “{ML-Doctor}: Holistic risk assessment of inference attacks against machine learning models,” in 31st USENIX Security Symposium (USENIX Security 22) , 2022, pp. 4525–4542

    2022

  48. [55]

    Feature inference attack on model predictions in vertical federated learning,

    X. Luo, Y . Wu, X. Xiao, and B. C. Ooi, “Feature inference attack on model predictions in vertical federated learning,” in 2021 IEEE 37th international conference on data engineering (ICDE). IEEE, 2021, pp. 181–192

    2021

  49. [56]

    Practical privacy attacks on vertical federated learning,

    H. Weng, J. Zhang, X. Ma, F. Xue, T. Wei, S. Ji, and Z. Zong, “Practical privacy attacks on vertical federated learning,” arXiv preprint arXiv:2011.09290, 2020

    2011 arXiv

  50. [57]

    Is vertical logistic regression privacy-preserving? a comprehensive privacy analysis and beyond,

    Y . Hu, T. Cai, J. Shan, S. Tang, C. Cai, E. Song, B. Li, and D. Song, “Is vertical logistic regression privacy-preserving? a comprehensive privacy analysis and beyond,”arXiv preprint arXiv:2207.09087, 2022

    2022 arXiv

  51. [58]

    Model inversion attacks against collaborative inference,

    Z. He, T. Zhang, and R. B. Lee, “Model inversion attacks against collaborative inference,” in Proceedings of the 35th Annual Computer Security Applications Conference , 2019, pp. 148–162

    2019

  52. [59]

    Comprehensive analysis of privacy leakage in vertical federated learning during prediction,

    X. Jiang, X. Zhou, and J. Grossklags, “Comprehensive analysis of privacy leakage in vertical federated learning during prediction,” Proceedings on privacy enhancing technologies, 2022

    2022

  53. [60]

    Cafe: Catastrophic data leakage in vertical federated learning,

    X. Jin, P.-Y . Chen, C.-Y . Hsu, C.-M. Yu, and T. Chen, “Cafe: Catastrophic data leakage in vertical federated learning,” Advances in neural information processing systems, vol. 34, pp. 994–1006, 2021

    2021

  54. [61]

    Defending batch- level label inference and replacement attacks in vertical federated learning,

    T. Zou, Y . Liu, Y . Kang, W. Liu, Y . He, Z. Yi, Q. Yang, and Y .-Q. Zhang, “Defending batch- level label inference and replacement attacks in vertical federated learning,”IEEE Transactions on Big Data, 2022

    2022

  55. [62]

    Exploit: Extracting private labels in split learning,

    S. Kariyappa and M. K. Qureshi, “Exploit: Extracting private labels in split learning,” in 2023 IEEE conference on secure and trustworthy machine learning (SaTML). IEEE, 2023, pp. 165–175

    2023

  56. [63]

    Residue-based label protection mechanisms in vertical logistic regression,

    J. Tan, L. Zhang, Y . Liu, A. Li, and Y . Wu, “Residue-based label protection mechanisms in vertical logistic regression,” in 2022 8th International Conference on Big Data Computing and Communications (BigCom). IEEE, 2022, pp. 356–364

    2022

  57. [64]

    See through gradi- ents: Image batch recovery via gradinversion,

    H. Yin, A. Mallya, A. Vahdat, J. M. Alvarez, J. Kautz, and P. Molchanov, “See through gradi- ents: Image batch recovery via gradinversion,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 16 337–16 346

    2021

  58. [65]

    Cgir: Conditional generative instance reconstruction attacks against federated learning,

    X. Xu, P. Liu, W. Wang, H.-L. Ma, B. Wang, Z. Han, and Y . Han, “Cgir: Conditional generative instance reconstruction attacks against federated learning,” IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 6, pp. 4551–4563, 2022

    2022

  59. [66]

    Feature reconstruction attacks and countermeasures of dnn training in vertical federated learning,

    P. Ye, Z. Jiang, W. Wang, B. Li, and B. Li, “Feature reconstruction attacks and countermeasures of dnn training in vertical federated learning,” IEEE Transactions on Dependable and Secure Computing, 2024

    2024

  60. [67]

    Label inference attacks against vertical federated learning,

    C. Fu, X. Zhang, S. Ji, J. Chen, J. Wu, S. Guo, J. Zhou, A. X. Liu, and T. Wang, “Label inference attacks against vertical federated learning,” in 31st USENIX security symposium (USENIX Security 22), 2022, pp. 1397–1414

    2022

  61. [68]

    Label leakage and protection in two-party split learning,

    O. Li, J. Sun, X. Yang, W. Gao, H. Zhang, J. Xie, V . Smith, and C. Wang, “Label leakage and protection in two-party split learning,” arXiv preprint arXiv:2102.08504, 2021

    2021 arXiv

  62. [69]

    Label leakage and protection from forward embedding in vertical federated learning,

    J. Sun, X. Yang, Y . Yao, and C. Wang, “Label leakage and protection from forward embedding in vertical federated learning,” arXiv preprint arXiv:2203.01451, 2022. 23

    2022 arXiv

  63. [70]

    Your labels are selling you out: Relation leaks in vertical federated learning,

    P. Qiu, X. Zhang, S. Ji, T. Du, Y . Pu, J. Zhou, and T. Wang, “Your labels are selling you out: Relation leaks in vertical federated learning,” IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 5, pp. 3653–3668, 2022

    2022

  64. [71]

    Federated learning with differential privacy: Algorithms and performance analysis,

    K. Wei, J. Li, M. Ding, C. Ma, H. H. Yang, F. Farokhi, S. Jin, T. Q. Quek, and H. V . Poor, “Federated learning with differential privacy: Algorithms and performance analysis,” IEEE transactions on information forensics and security, vol. 15, pp. 3454–3469, 2020

    2020

  65. [72]

    Ldp-fed: Federated learning with local differential privacy,

    S. Truex, L. Liu, K.-H. Chow, M. E. Gursoy, and W. Wei, “Ldp-fed: Federated learning with local differential privacy,” inProceedings of the third ACM international workshop on edge systems, analytics and networking, 2020, pp. 61–66

    2020

  66. [73]

    Safelearn: Secure aggregation for private federated learning,

    H. Fereidooni, S. Marchal, M. Miettinen, A. Mirhoseini, H. Möllering, T. D. Nguyen, P. Rieger, A.-R. Sadeghi, T. Schneider, H. Yalame et al., “Safelearn: Secure aggregation for private federated learning,” in 2021 IEEE Security and Privacy Workshops (SPW). IEEE, 2021, pp. 56–62

    2021

  67. [74]

    Lightsecagg: a lightweight and versatile design for secure aggregation in federated learning,

    J. So, C. He, C.-S. Yang, S. Li, Q. Yu, R. E Ali, B. Guler, and S. Avestimehr, “Lightsecagg: a lightweight and versatile design for secure aggregation in federated learning,” Proceedings of Machine Learning and Systems, vol. 4, pp. 694–720, 2022

    2022

  68. [75]

    {BatchCrypt}: Efficient homomorphic encryption for {Cross-Silo} federated learning,

    C. Zhang, S. Li, J. Xia, W. Wang, F. Yan, and Y . Liu, “{BatchCrypt}: Efficient homomorphic encryption for {Cross-Silo} federated learning,” in 2020 USENIX annual technical conference (USENIX ATC 20), 2020, pp. 493–506

    2020

  69. [76]

    Homomorphic encryption-based privacy-preserving federated learning in iot-enabled healthcare system,

    L. Zhang, J. Xu, P. Vijayakumar, P. K. Sharma, and U. Ghosh, “Homomorphic encryption-based privacy-preserving federated learning in iot-enabled healthcare system,”IEEE Transactions on Network Science and Engineering, vol. 10, no. 5, pp. 2864–2880, 2022

    2022

  70. [78]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,

    S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149, 2015

    2015 arXiv

  71. [79]

    Fedmd: Heterogenous federated learning via model distillation,

    D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,”arXiv preprint arXiv:1910.03581, 2019

    1910 arXiv

  72. [80]

    Communication-efficient on- device machine learning: Federated distillation and augmentation under non-iid private data,

    E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Communication-efficient on- device machine learning: Federated distillation and augmentation under non-iid private data,” arXiv preprint arXiv:1811.11479, 2018

    2018 arXiv

  73. [81]

    Split learning for health: Distributed deep learning without sharing raw patient data,

    P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar, “Split learning for health: Distributed deep learning without sharing raw patient data,” arXiv preprint arXiv:1812.00564, 2018

    2018 arXiv

  74. [82]

    Joint horizontal and vertical federated learning for multimodal iot,

    Y . Peng, Z. Lu, and J. Xu, “Joint horizontal and vertical federated learning for multimodal iot,” in Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, 2024, pp. 2371–2376

    2024

  75. [83]

    Sparse communication for distributed gradient descent,

    A. F. Aji and K. Heafield, “Sparse communication for distributed gradient descent,” arXiv preprint arXiv:1704.05021, 2017

    2017 arXiv

  76. [84]

    Qsgd: Communication-efficient sgd via gradient quantization and encoding,

    D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. V ojnovic, “Qsgd: Communication-efficient sgd via gradient quantization and encoding,” Advances in neural information processing systems, vol. 30, 2017

    2017

  77. [85]

    Fedpaq: A communication-efficient federated learning method with periodic averaging and quantiza- tion,

    A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, “Fedpaq: A communication-efficient federated learning method with periodic averaging and quantiza- tion,” in International conference on artificial intelligence and statistics. PMLR, 2020, pp. 2021–2031

    2020

  78. [86]

    Asynchronous federated optimization,

    C. Xie, S. Koyejo, and I. Gupta, “Asynchronous federated optimization,” arXiv preprint arXiv:1903.03934, 2019. 24

    1903 arXiv

  79. [87]

    Client-edge-cloud hierarchical federated learning,

    L. Liu, J. Zhang, S. Song, and K. B. Letaief, “Client-edge-cloud hierarchical federated learning,” in ICC 2020-2020 IEEE international conference on communications (ICC). IEEE, 2020, pp. 1–6

    2020

  80. [88]

    Towards personalized federated learning,

    A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” IEEE transactions on neural networks and learning systems, vol. 34, no. 12, pp. 9587–9603, 2022

    2022

  81. [89]

    Fedmekt: Distillation-based embedding knowledge transfer for multimodal federated learning,

    H. Q. Le, M. N. Nguyen, C. M. Thwal, Y . Qiao, C. Zhang, and C. S. Hong, “Fedmekt: Distillation-based embedding knowledge transfer for multimodal federated learning,” Neural Networks, vol. 183, p. 107017, 2025

    2025

  82. [90]

    Fedcmd: A federated cross-modal knowledge distillation for drivers’ emotion recognition,

    S. Bano, N. Tonellotto, P. Cassarà, and A. Gotta, “Fedcmd: A federated cross-modal knowledge distillation for drivers’ emotion recognition,”ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1–27, 2024

    2024

  83. [91]

    Deep learning model with multi-feature fusion and label association for suicide detection,

    Z. Li, W. Cheng, J. Zhou, Z. An, and B. Hu, “Deep learning model with multi-feature fusion and label association for suicide detection,” Multimedia systems, vol. 29, no. 4, pp. 2193–2203, 2023

    2023

  84. [92]

    Embodied intelligence via learning and evolution,

    A. Gupta, S. Savarese, S. Ganguli, and L. Fei-Fei, “Embodied intelligence via learning and evolution,” Nature communications, vol. 12, no. 1, p. 5721, 2021

    2021

  85. [93]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervi- sion,” in International conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  86. [94]

    Knowledge-driven egocentric multimodal activity recognition,

    Y . Huang, X. Yang, J. Gao, J. Sang, and C. Xu, “Knowledge-driven egocentric multimodal activity recognition,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 16, no. 4, pp. 1–133, 2020

    2020

  87. [95]

    The kinetics human action video dataset,

    W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsevet al., “The kinetics human action video dataset,” arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  88. [96]

    Ucf101: A dataset of 101 human actions classes from videos in the wild,

    K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  89. [97]

    Human fall detection on embedded platform using depth maps and wireless accelerometer,

    B. Kwolek and M. Kepski, “Human fall detection on embedded platform using depth maps and wireless accelerometer,” Computer methods and programs in biomedicine, vol. 117, no. 3, pp. 489–501, 2014

    2014

  90. [98]

    Emotion recognition from multiple modalities: Fundamentals and methodologies,

    S. Zhao, G. Jia, J. Yang, G. Ding, and K. Keutzer, “Emotion recognition from multiple modalities: Fundamentals and methodologies,” IEEE Signal Processing Magazine, vol. 38, no. 6, pp. 59–73, 2021

    2021

  91. [99]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  92. [100]

    M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues,

    T. Mittal, U. Bhattacharya, R. Chandra, A. Bera, and D. Manocha, “M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 02, 2020, pp. 1359–1367

    2020

  93. [101]

    Meld: A multimodal multi-party dataset for emotion recognition in conversations,

    S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “Meld: A multimodal multi-party dataset for emotion recognition in conversations,” arXiv preprint arXiv:1810.02508, 2018

    2018 arXiv

  94. [102]

    Multimodal language anal- ysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,

    A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language anal- ysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2018

  95. [103]

    Vilt: Vision-and-language transformer without convolution or region supervision,

    W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in International conference on machine learning . PMLR, 2021, pp. 5583–5594. 25

    2021

  96. [104]

    Vlmo: Unified vision-language pre-training with mixture-of-modality-experts,

    H. Bao, W. Wang, L. Dong, Q. Liu, O. K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei, “Vlmo: Unified vision-language pre-training with mixture-of-modality-experts,” Advances in Neural Information Processing Systems, vol. 35, pp. 32 897–32 912, 2022

    2022

  97. [105]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,

    B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2641–2649

    2015

  98. [106]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” inComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13. Springer, 201...

    2014

  99. [107]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6904–6913

    2017

  100. [108]

    Client selection for federated learning with heterogeneous re- sources in mobile edge,

    T. Nishio and R. Yonetani, “Client selection for federated learning with heterogeneous re- sources in mobile edge,” in IEEE International Conference on Communications (ICC), 2019

    2019

  101. [109]

    Splitfed: When federated learning meets split learning,

    Y . Jiang, J. Koneˇcn`y, S. Tople, F. Wang, D. Beutel, B. Zhaoet al., “Splitfed: When federated learning meets split learning,” in International Conference on Machine Learning (ICML) , 2023

    2023

  102. [110]

    How to backdoor federated learning,

    E. Bagdasaryan, A. Veit, Y . Hua, D. Estrin, and V . Shmatikov, “How to backdoor federated learning,” in Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), 2020

    2020

  103. [111]

    Model inversion attacks that exploit confidence information and basic countermeasures,

    M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, 2015, pp. 1322–1333

    2015

  104. [112]

    Deep models under the gan: Information leakage from collaborative deep learning,

    B. Hitaj, G. Ateniese, and F. Perez-Cruz, “Deep models under the gan: Information leakage from collaborative deep learning,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2017

    2017

  105. [113]

    Privacy-preserving deep learning via additively homomorphic encryption,

    Y . Aono, T. Hayashi, M. Abuhamadet al., “Privacy-preserving deep learning via additively homomorphic encryption,” in IEEE Transactions on Information Forensics and Security, 2017

    2017

  106. [114]

    Differentially private federated learning: A client level perspective,

    R. C. Geyer, T. Klein, and M. Nabi, “Differentially private federated learning: A client level perspective,” in NeurIPS Workshop on Private Multi-Party Machine Learning, 2017

    2017

  107. [115]

    Blockchain federated learning: A robust decentralized learning framework,

    C. Zhang, S. Xie, H. Bai, F. Yuet al., “Blockchain federated learning: A robust decentralized learning framework,” IEEE Transactions on Industrial Informatics, 2020

    2020

  108. [116]

    Federated transfer learning with multimodal data,

    Y . Sun, “Federated transfer learning with multimodal data,”arXiv preprint arXiv:2209.03137, 2022

    2022 arXiv

  109. [117]

    Federated self-supervised learning of multisensor representations for embedded intelligence,

    A. Saeed, F. D. Salim, T. Ozcelebi, and J. Lukkien, “Federated self-supervised learning of multisensor representations for embedded intelligence,” IEEE Internet of Things Journal , vol. 8, no. 2, pp. 1030–1040, 2020

    2020

  110. [118]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning. PmLR, 2020, pp. 1597–1607

    2020

  111. [119]

    Contrastive multiview coding,

    Y . Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” inEuropean Conference on Computer Vision (ECCV), 2020

    2020

  112. [120]

    Federated semi-supervised learning with inter-client consistency and disjoint learning,

    E. Jeong, J. Shin, and J. Kim, “Federated semi-supervised learning with inter-client consistency and disjoint learning,” in International Conference on Learning Representations (ICLR), 2020

    2020

  113. [121]

    Federated learning with person- alization layers,

    M. Arivazhagan, A. Aggarwal, N. Singh, and S. Choudhary, “Federated learning with person- alization layers,” in NeurIPS Workshop on Federated Learning, 2019. 26

    2019

  114. [122]

    Federated multi-task learning,

    V . Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, “Federated multi-task learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  115. [123]

    Towards personalized federated learning via heterogeneous model reassembly,

    J. Wang, X. Yang, S. Cui, L. Che, L. Lyu, D. D. Xu, and F. Ma, “Towards personalized federated learning via heterogeneous model reassembly,” Advances in Neural Information Processing Systems, vol. 36, pp. 29 515–29 531, 2023

    2023

  116. [124]

    Fedkd: Communication-efficient federated learning via knowledge distillation,

    H. Zhang, Y . Wang, Y . Li, Z. Liet al., “Fedkd: Communication-efficient federated learning via knowledge distillation,” in IEEE International Conference on Data Mining (ICDM), 2022

    2022

  117. [125]

    Gpt-4o technical report,

    OpenAI, “Gpt-4o technical report,” 2024, https://openai.com/index/gpt-4o

    2024

  118. [126]

    Grad-cam: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. 27

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.