Pith. sign in

REVIEW 3 major objections 6 minor 82 references

Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MR

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Combining multi-modal foundation models with federated learning can deliver privacy-preserving, personalized intelligence inside AR/VR/MR systems, and the paper organizes the obstacles around five SHIFT dimensions.

desk verdict A well-organized XR+FL roadmap whose SHIFT taxonomy is genuinely useful, but the core feasibility claim is asserted rather than demonstrated; still deserves peer review as a perspective. read the letter →

arxiv 2506.05683 v4 pith:FUR6KSRM submitted 2025-06-06 cs.LG cs.AIcs.CRcs.MM

Fardis Nadimi , Payam Abdisarabshali , Kasra Borazjani , Jacob Chakareski , Seyyedali Hosseinalipour This is my paper
classification cs.LGcs.AIcs.CRcs.MM
keywords federatedlearningfoundationmodelsextendedrealitymulti-modalpersonalizationon-deviceAIprivacySHIFTdimensions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This perspective paper argues that combining multi-modal multi-task foundation models with federated learning—forming what it calls M3T federated foundation models (FedFMs)—is the right path to privacy-preserving, personalized intelligence in extended reality (XR). The paper's central contribution is a structured map of the field: a modular model architecture, three coordination paradigms for training and aggregation, and a five-part taxonomy of XR-specific challenges called SHIFT (sensor/modality diversity, hardware heterogeneity, interactivity and embodied personalization, functional/task variability, temporality/environmental variability). It also proposes evaluation metrics, dataset requirements, and design tradeoffs for building resource-aware FedFMs. A sympathetic reader would care because XR devices continuously collect deeply personal embodied data, and the paper offers a concrete agenda for keeping that data on-device while still benefiting from large shared models.

What carries the argument

The argument is carried by a modular architecture for M3T FedFMs: modality encoders turn raw sensory streams into embeddings; a shared backbone (transformer, dual-encoder, or mixture-of-experts) performs fusion and reasoning; task heads map representations to outputs; and lightweight adapters or prompts enable local personalization. Around this sits the federated coordination layer, with centralized, fully decentralized, and semi-decentralized aggregation paradigms. The enabling assumption is that parameter-efficient fine-tuning (for example, low-rank adaptation) lets each device update only small modules while the global backbone stays shared. The SHIFT taxonomy is the paper's organizing device: it turns the broad question 'what is hard about FedFMs in XR?' into five named dimensions against which every proposed solution can be evaluated.

What would settle it

Run a controlled experiment on a consumer XR headset: fine-tune a compact multi-modal foundation model locally using LoRA on a live gaze/gesture stream while tracking per-interaction latency and a multi-task accuracy benchmark. If per-update latency exceeds roughly 50 ms or accuracy falls materially below centralized fine-tuning, the paper's feasibility premise fails; if the global model diverges under simulated modality dropout during federated aggregation, the SHIFT sensor-diversity claim is confirmed as the key obstacle.

Watch

Extended reading notes

Core claim

The paper's central claim is that the seamless integration of M3T FedFMs into XR systems is both desirable and increasingly feasible, and that the obstacles to it are not random engineering noise but a coherent set of dimensions that can be named, measured, and addressed. It codifies these as the SHIFT dimensions and shows how they manifest across applications such as cognitive XR overlays, neuroadaptive interfaces, embodied AI companions, and emotion-aware storytelling. The paper further claims that federated learning resolves the privacy, bandwidth, and personalization barriers of centralized foundation-model training, while parameter-efficient fine-tuning (adapter, prompt, and LoRA-style updates) makes on-device adaptation plausible. The intended outcome is a research framework: anyone building FedFMs for XR can use the architecture, metrics, tradeoffs, and dataset list to position their work.

Load-bearing premise

The load-bearing premise is that compact multi-modal foundation models, together with parameter-efficient fine-tuning such as LoRA and adapters, can be trained and run on XR-class hardware at interactive latency without losing the multi-modal, multi-task capabilities that make foundation models useful.

Editorial extensions

If this is right

  • If on-device PEFT works as assumed, XR devices can personalize large shared models without uploading gaze, motion, or biometric data to a central server.
  • The SHIFT dimensions give researchers a common vocabulary and a checklist: a proposed FedFM method can be tested against each dimension (missing modalities, hardware asymmetry, interactivity, task variability, temporal drift).
  • The evaluation metrics (inference latency, personalization effectiveness, submodule activation efficiency, communication overhead, privacy leakage risk) could become the standard benchmark suite for XR FedFMs.
  • The dataset list identifies concrete gaps—federated XR simulators, role-specific benchmarks, longitudinal interaction logs, cognitive-load-aware datasets—that the community would need to build before real progress can be measured.
  • Semi-decentralized aggregation offers a practical middle path: cluster-level D2D consensus for personalization plus a single uplink per cluster for global consistency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The SHIFT taxonomy likely generalizes beyond XR: any federated edge setting with heterogeneous sensors, strict latency, and personalization—such as robot fleets or in-car assistants—could adopt the same five dimensions, though the paper does not make this claim.
  • A testable extension would be to simulate a FedFM with randomly masked modalities per client and measure convergence; the paper predicts this will degrade naive aggregation but does not quantify how.
  • The privacy argument is conditional: model updates and gradient information can still leak sensitive information, so the paper's vision implicitly requires differential privacy or secure aggregation despite treating privacy as a solved benefit of FL.
  • One could benchmark whether sub-50ms interaction latency is actually achievable when prompt/adapters are updated on-device while the backbone runs; the paper cites feasibility but stops short of a system measurement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This perspective paper envisions integrating multi-modal multi-task federated foundation models (M3T FedFMs) into extended reality (XR) systems as a route to privacy-preserving, personalized, distributed intelligence. It proposes a modular FM architecture (modality encoders, shared backbone with transformer/dual-encoder/MoE variants, task heads, adapters/prompts), three FL coordination paradigms (centralized, fully decentralized, semi-decentralized), and a taxonomy of XR-specific challenges organized into the SHIFT dimensions: sensor and modality diversity, hardware heterogeneity, interactivity and embodied personalization, functional/task variability, and temporality/environmental variability. The paper illustrates SHIFT across application scenarios (perceptual overlays, neuroadaptive interfaces, embodied AI companions, emotion-aware gaming), then proposes evaluation metrics mapped to SHIFT, a dataset gap analysis, design tradeoffs, and five sets of high-level solution directions. It contains no experiments, derivations, or quantitative support; it explicitly describes itself as a perspective and defers technical investigation to future work. The feasibility of on-device M3T FedFMs is argued from compact on-device models (Gemma 3n, Llama 3.2, Apple Intelligence) and parameter-efficient fine-tuning (LoRA, adapters, prompts).

Significance. If its roadmap is executable, this paper would provide a useful organizing framework for a young research area: the SHIFT taxonomy is coherent and each dimension is instantiated with concrete XR phenomena; Table 6 maps evaluation metrics to measurement methods and SHIFT dimensions; Table 8 usefully separates existing datasets from ones that need to be created; and the scenarios connect the framework to plausible applications. The paper is candid about gaps, explicitly stating that extending PEFT, distillation, quantization, and client selection to modular M3T FedFMs remains highly underexplored and that on-device fine-tuning strategies may fall short in XR contexts. No code, experiments, or falsifiable predictions accompany the paper, so the value is conditional on whether the asserted feasibility premise (that compact M3T FMs with PEFT can be fine-tuned on XR-class devices at interactive latency) actually holds. That premise is the weakest load-bearing point: the cited evidence concerns on-device inference, not federated fine-tuning, whose training-time memory, energy, and thermal costs are materially higher.

major comments (3)
  1. [What Signals Point to the Feasibility of Integrating M3T FedFMs into the XR Ecosystem?] The feasibility case rests on a mismatch between the evidence and the claim it supports. The paper asserts that FedFMs are becoming increasingly practical and cites Gemma 3n, Llama 3.2, and Apple Intelligence, noting these are specifically engineered for real-time, on-device multi-modal inference. On-device inference is not the same as on-device federated fine-tuning: local training with LoRA or adapters requires backpropagation, activation storage, optimizer states, and repeated local epochs, whose memory, energy, thermal, and latency costs materially exceed inference, and the paper neither quantifies nor identifies these costs. The paper's own Training Strategies paragraph concedes that extending PEFT, distillation, quantization, and client selection to modular M3T FedFMs remains highly underexplored, and Dimension 3 concedes that the fine-tuning strategies may fall short in XR contexts as too resource-intensive or insufficiently responsive. The headline vision therefore rests on an assumption the cited sources do not establish. The revision should either re-scope the feasibility language to separate on-device inference from federated fine-tuning, or provide a concrete account of the training-time resource envelope (memory, energy, thermal, latency) implied by local modular fine-tuning on XR-class hardware; as written, the text presents an asserted premise as if it were evidenced.
  2. [FL Coordination Paradigm and Training Strategies of M3T FedFMs] The central operational step of the proposed paradigm (aggregation) is left unspecified for exactly the regime the paper identifies as hardest. The centralized paradigm is described as aggregation typically via weighted averaging, and the training-strategies paragraph calls for modular-level aggregations, but no mechanism is given for averaging or otherwise combining parameters when clients hold non-identical parameter spaces (different adapters, prompts, expert modules, or task heads). This is not a minor detail: Table 3 itself states that standard aggregation methods assume homogeneous models and data distributions and are ineffective with heterogeneous client models and tasks, but no candidate aggregation space is offered as a replacement. A concrete sketch is available in principle (for example, federated averaging of LoRA delta matrices over a shared frozen backbone, module-type-clustered aggregation, or prompt/head ensembles), and the paper should either provide such a sketch or explicitly elevate modular aggregation to the status of the central open problem of M3T FedFM research, with the convergence-theoretic limitations acknowledged. The Solutions section legitimately scopes out detailed technical work, but the architecture and training sections still assert the aggregation step, so the open-problem status needs to be stated where the assertion is made.
  3. [Why in the Above (and All Comparable) Scenarios M3T FMs Must be Accompanied by FL?] The headline promise is privacy-preserving distributed intelligence, and the paper claims that sharing only model/gradients parameters prevents raw user data (e.g., emotional states, biometric signals) from being exposed to external entities. This is stronger than the paper's own Table 3, which notes that model updates can still leak sensitive information unless security countermeasures (e.g., differential privacy or encryption) are used. Yet none of the SHIFT dimensions, solution directions, or evaluation metrics include those countermeasures: there is no treatment of differential privacy, secure aggregation, gradient inversion, membership inference, or the privacy-related tradeoffs that Table 7 gestures at. For a paper whose title and central claim are privacy-focused, the absence of a privacy-risk agenda (despite the in-text acknowledgment that FL alone leaks) is a substantive omission in the roadmap. The revision should add privacy-preservation mechanisms to the research agenda or explicitly scope them out of the paper's focus; as written, the strong privacy claim and the roadmap's content are in tension.
minor comments (6)
  1. [Abstract] The abstract states that XR consist of virtual reality (VR), augmented reality (AR), and mixed reality (XR); the parenthetical should read (MR).
  2. [Manifestations of SHIFT Dimensions] The header for Dimension 1 appears as S ensor and Modality Diversity with an extra space in Sensor, and F unctional/Task Variability has the same artifact; these formatting errors should be cleaned up.
  3. [What Signals Point to the Feasibility of Integrating M3T FedFMs into the XR Ecosystem?] The sentence FedFMs and M3T FedFMs are becoming increasingly practical uses both the abbreviated and full forms immediately after the paper announces that it will use the shorter abbreviation exclusively; pick one form.
  4. [What Signals Point to the Feasibility of Integrating M3T FedFMs into the XR Ecosystem?] Reference [28], the authors' own position paper, is cited as evidence that FedFMs are becoming increasingly practical; a self-citation of a position paper does not constitute practical evidence and should be supplemented by citations to implemented systems or benchmark studies.
  5. [Table 6] The Personalization Necessity Rate metric fixes a threshold of baseline accuracy (e.g., 90%) without justification; the threshold should be presented as an experimental parameter that varies across applications rather than as a fixed criterion.
  6. [Manifestations of SHIFT Dimensions] The note that the SHIFT dimensions are complementary to those of conventional FL (e.g., data heterogeneity) would benefit from an explicit statement of which phenomena are new to XR versus intensified versions of known FL challenges (hardware heterogeneity and task variability, for instance, are standard FL topics), so that the reader can see where the claimed novelty lies.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper is a perspective with no fitted predictions; one minor self-citation is not load-bearing.

full rationale

This is a perspective/vision paper, not an empirical derivation. It contains no fitted parameters, no equations, and no quantitative prediction that could reduce to an input by construction. The central contribution is the SHIFT taxonomy plus evaluation metrics, dataset proposals, and design tradeoffs, which are organizational and forward-looking rather than derived from the cited literature. The only self-referential element is the sentence in the feasibility discussion that cites the authors' own prior M3T FedFM position paper (ref 28) as evidence that 'FedFMs and M3T FedFMs are becoming increasingly practical.' This is a mild self-citation, but it is not load-bearing: the same sentence independently cites refs 29-30 and on-device model developments 31-33, and the paper's XR-specific contribution does not depend on ref 28 as a premise. Separately, the concern that on-device inference results do not establish on-device federated fine-tuning costs is a systems feasibility risk, not circularity, because the paper explicitly marks the extension of PEFT and related techniques to modular M3T FedFMs as 'highly underexplored' and positions itself as a call for future research. Therefore, no circular reduction is exhibited.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central vision rests on feasibility assumptions about on-device FMs, the necessity of FL for XR data, and the transferability of existing FL and PEFT methods to modular M3T models. These are reasonable but unproven premises, and the paper itself acknowledges that detailed technical investigation lies outside its scope.

assumptions (3)
  • domain assumption Compact M3T foundation models and parameter-efficient fine-tuning can run on XR-class devices at interactive latency without losing multi-modal multi-task capabilities.
    Invoked in the feasibility section citing Gemma 3n, Llama 3.2, and Apple Intelligence as evidence that on-device FMs are becoming practical; no XR-specific resource budget is analyzed.
  • domain assumption Raw XR data cannot be centralized because of privacy, bandwidth, and personalization constraints, making federated training necessary.
    Stated in the section 'Why in the Above Scenarios M3T FMs Must be Accompanied by FL?' and treated as a fixed constraint rather than a design choice that could be traded off.
  • domain assumption Existing FL aggregation and parameter-efficient fine-tuning methods extend naturally to modular, multi-modal multi-task architectures.
    The section 'Training Strategies of M3T FedFMs' notes that the extension is 'highly underexplored', which is an admission that this is an open assumption rather than an established fact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MR." pith.science (2026). https://pith.science/paper/FUR6KSRM

@misc{pith2026250605683,
  author       = {Pith},
  title        = {Pith review of: Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FUR6KSRM}},
  note         = {Machine review of arXiv:2506.05683}
}
read the original abstract

Extended reality (XR) systems, which consist of virtual reality (VR), augmented reality (AR), and mixed reality (XR), offer a transformative interface for immersive, multi-modal, and embodied human-computer interaction. In this paper, we envision that multi-modal multi-task (M3T) federated foundation models (FedFMs) can offer transformative capabilities for XR systems through integrating the representational strength of M3T foundation models (FMs) with the privacy-preserving model training principles of federated learning (FL). We present a modular architecture for FedFMs, which entails different coordination paradigms for model training and aggregations. Central to our vision is the codification of XR challenges that affect the implementation of FedFMs under the SHIFT dimensions: (1) Sensor and modality diversity, (2) Hardware heterogeneity and system-level constraints, (3) Interactivity and embodied personalization, (4) Functional/task variability, and (5) Temporality and environmental variability. We illustrate the manifestation of these dimensions across a set of emerging and anticipated applications of XR systems. Finally, we propose evaluation metrics, dataset requirements, and design tradeoffs necessary for the development of resource-aware FedFMs in XR. This perspective aims to chart the technical and conceptual foundations for context-aware privacy-preserving intelligence in the next generation of XR systems.

Figures

Figures reproduced from arXiv: 2506.05683 by the authors.

Figure 1
Figure 1. Various types of XR: (a) Augmented Reality, where digital elements are embedded into the environment surrounding a user, (b) Virtual Reality, where a user is fully immersed into a completely synthetic environment and interacts with various elements therein, and (c) Mixed Reality, which blends digital and physical worlds where a user can interact with digital objects. of downstream tasks (e.g., text/image/audio gener… view at source ↗
Figure 2
Figure 2. Foundation model architecture containing the input layer (i.e., modality encoders), the FM backbone, and the output layer (i.e., task heads). The FM backbone can comprise varying architectures, such as Mixture-of-Experts (MoE), GPT-style (stacked transformers), and CLIP-style (contrastive learning-based, where transformers are used for image and text processing) architectures. forms are depicted at the bottom of [P… view at source ↗
Figure 3
Figure 3. Various model aggregation and network coordination strategies in FL: (a) Conventional FL employs a centralized model aggregation strategy where clients perform local training and send their model updates to a central server. The server aggregates these updates into a global model and broadcasts it back to the clients. (b) Semi-decentralized FL introduces a hierarchical structure where clients are organized into clus… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: FedFM deployment visualization in an XR system, under FL with centralized model aggregation strategy. Different deployment environments of XR devices will result in varying tasks and modalities across the clients. a paralyzed limb. Subsequently, the M3T FM, trained to …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

82 extracted references · 67 canonical work pages

  1. [1]

    A., Felix, R., Hinsch, C., Shahab, H

    Rauschnabel, P. A., Felix, R., Hinsch, C., Shahab, H. & Alt, F. What is XR? Towards a framework for augmented and virtual reality. Comput. human behavior 133, 107289 (2022)

    2022

  2. [2]

    Virtual, augmented and mixed reality are the 4th wave

    Merel, T. Virtual, augmented and mixed reality are the 4th wave. The Wrap News Inc. (2016). 14/18

    2016

  3. [3]

    Virtual and augmented reality market size research reports

    Fortune Business Insights. Virtual and augmented reality market size research reports. [Online] (2023). https://www. fortunebusinessinsights.com/industry-reports/virtual-reality-market-101378, https://www.fortunebusinessinsights.com/ augmented-reality-ar-market-102553

    work page 2023

  4. [4]

    A comprehensive review of multimodal XR applications, risks, and ethical challenges in the metaverse

    Kourtesis, P. A comprehensive review of multimodal XR applications, risks, and ethical challenges in the metaverse. Multimodal Technol. Interact. 8, 98 (2024)

    work page 2024

  5. [5]

    & Dodds, H

    Ziker, C., Truman, B. & Dodds, H. Cross reality (xr): Challenges and opportunities across the spectrum. Innov. learning environments STEM higher education: Oppor. challenges, looking forward 55–77 (2021)

    work page 2021

  6. [6]

    & Khan, M

    Chakareski, J. & Khan, M. Live 360◦ video streaming to heterogeneous clients in 5G networks. IEEE Trans. Multimed. 26, 8860–8873 (2024)

    work page 2024

  7. [7]

    & Blandino, S

    Chakareski, J., Khan, M., Ropitault, T. & Blandino, S. Millimeter wave and free-space-optics for future dual-connectivity 6DOF mobile multi-user VR streaming. ACM Trans. Multimed. Comput. Commun. Appl. 19, 57:1–25 (2023)

    work page 2023

  8. [8]

    & Popovski, P

    Gupta, S., Chakareski, J. & Popovski, P. mmWave networking and edge computing for scalable 360-degree video multi-user virtual reality. IEEE Trans. Image Process.32, 377–391 (2023)

    work page 2023

Show all 82 references
  1. [9]

    & Hashemi, M

    Badnava, B., Chakareski, J. & Hashemi, M. Neural-enhanced rate adaptation and computation distribution for emerging mmWave multi-user 3D video streaming systems. IEEE Trans. Multimed. (2025). Accepted

    2025

  2. [10]

    Wang, Z., Rao, M., Ye, S., Song, W. & Lu, F. Towards spatial computing: Recent advances in multimodal natural interaction for XR headsets. arXiv preprint arXiv:2502.07598 (2025)

    2025 arXiv

  3. [11]

    Bommasani, R. et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)

    2021 arXiv

  4. [12]

    Yang, L. et al. Diffusion models: A comprehensive survey of methods and applications. ACM Comput. Surv. 56, 1–39 (2023)

    2023

  5. [13]

    Naveed, H. et al. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435 (2023)

    2023 arXiv

  6. [14]

    Luo, S. et al. Delving into multi-modal multi-task foundation models for road scene understanding: From learning paradigm perspectives. IEEE Transactions on Intell. Veh.(2024)

    2024

  7. [15]

    & y Arcas, B

    McMahan, B., Moore, E., Ramage, D., Hampson, S. & y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, 1273–1282 (PMLR, 2017)

    2017

  8. [16]

    Towards artificial general or personalized intelligence? A survey on foundation models for personalized federated intelligence

    Qiao, Y .et al. Towards artificial general or personalized intelligence? A survey on foundation models for personalized federated intelligence. arXiv preprint arXiv:2505.06907 (2025)

    2025 arXiv

  9. [17]

    & Tresp, V

    Chen, H., Zhang, Y ., Krompass, D., Gu, J. & Tresp, V . FedDAT: An approach for foundation model finetuning in multi-modal heterogeneous federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 11285–11293 (2024)

    2024

  10. [18]

    Wang, T., Du, Y ., Gong, Y ., Choo, K.-K. R. & Guo, Y . Applications of federated learning in mobile health: Scoping review. J. Med. Internet Res. 25, e43006 (2023)

    2023

  11. [19]

    & Rampa, V

    Savazzi, S., Nicoli, M. & Rampa, V . Federated learning with cooperating devices: A consensus approach for massive IoT networks. IEEE Internet Things J. 7, 4641–4654 (2020)

    2020

  12. [20]

    Nguyen, D. C. et al. Federated learning for industrial internet of things in future industries. IEEE Wirel. Commun. 28, 192–199 (2021)

    2021

  13. [21]

    M., Soner, B., C ¸¨oleri, S., G ¨und¨uz, D

    Elbir, A. M., Soner, B., C ¸¨oleri, S., G ¨und¨uz, D. & Bennis, M. Federated learning in vehicular networks. In IEEE Int. Mediterranean Conf. Commun. Netw. (MeditCom), 72–77 (IEEE, 2022)

    2022

  14. [22]

    J., Chiang, M

    Han, D.-J., Hosseinalipour, S., Love, D. J., Chiang, M. & Brinton, C. G. Cooperative federated learning over hybrid terrestrial and non-terrestrial networks. In IEEE Int. Conf. Commun. (ICC), 2992–2997 (IEEE, 2024)

    2024

  15. [23]

    Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259–265 (2023)

    2023

  16. [24]

    Afzal, M. Z. et al. Next generation XR systems-large language models meet augmented and virtual reality. IEEE computer graphics applications (2025)

    2025

  17. [25]

    & Wang, F.-Y

    Li, X., Tian, Y ., Ye, P., Duan, H. & Wang, F.-Y . A novel scenarios engineering methodology for foundation models in metaverse. IEEE Transactions on Syst. Man, Cybern. Syst. 53, 2148–2159 (2022)

    2022

  18. [26]

    Reality Labs

    Meta. Reality Labs. https://tech.facebook.com/reality-labs/ (2025). Accessed: May, 2025

    2025

  19. [27]

    Omniverse

    NVIDIA. Omniverse. https://www.nvidia.com/en-us/omniverse/ (2025). Accessed: May, 2025. 15/18

    2025

  20. [28]

    Borazjani, K. et al. Multi-modal multi-task (M3T) federated foundation models for embodied AI: Potentials and challenges for edge integration. arXiv preprint arXiv:2505.11191 (2025)

    2025

  21. [29]

    Du, J. et al. Distributed foundation models for multi-modal learning in 6G wireless networks. IEEE Wirel. Commun. 31, 20–30 (2024)

    2024

  22. [30]

    & Zhang, W

    Li, X., Peng, L., Wang, Y .-P. & Zhang, W. Open challenges and opportunities in federated foundation models towards biomedical healthcare. BioData Min. 18, 2 (2025)

    2025

  23. [31]

    Announcing gemma 3n preview: powerful, efficient, mobile-first ai

    Google. Announcing gemma 3n preview: powerful, efficient, mobile-first ai. https://developers.googleblog.com/en/ introducing-gemma-3n (2025)

    2025

  24. [32]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/ (2024)

    2024

  25. [33]

    Introducing apple’s on-device and server foundation models

    Apple. Introducing apple’s on-device and server foundation models. https://machinelearning.apple.com/research/ introducing-apple-foundation-models (2025)

    2025

  26. [34]

    & Lyu, L

    Zhuang, W., Chen, C. & Lyu, L. When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv preprint arXiv:2306.15546 (2023)

    2023 arXiv

  27. [35]

    & Zhang, A

    Chen, J. & Zhang, A. On disentanglement of asymmetrical knowledge transfer for modality-task agnostic federated learning. In Proc. AAAI Conf. Artificial Intell., vol. 38, 11311–11319 (2024)

    2024

  28. [36]

    Xiao, C. et al. Configurable foundation models: Building LLMs from a modular perspective. arXiv preprint arXiv:2409.02877 (2024)

    2024 arXiv

  29. [37]

    Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020)

    2020

  30. [38]

    Radford, A. et al. Learning transferable visual models from natural language supervision. In Int. Conf. Machine Learn., 8748–8763 (PmLR, 2021)

    2021

  31. [39]

    Long, G., Shen, T., Jiang, J., Blumenstein, M. et al. Dual-personalizing adapter for federated foundation models. Adv. Neural Inf. Process. Syst. 37, 39409–39433 (2024)

    2024

  32. [40]

    Jia, M. et al. Visual prompt tuning. In European conference on computer vision, 709–727 (Springer, 2022)

    2022

  33. [41]

    E., Alizadeh, M

    Ghiasvand, S., Oskouie, H. E., Alizadeh, M. & Pedarsani, R. Few-shot adversarial low-rank fine-tuning of vision-language models. arXiv preprint arXiv:2505.15130 (2025)

    2025

  34. [42]

    & Brinton, C

    Parasnis, R., Hosseinalipour, S., Chu, Y .-W., Chiang, M. & Brinton, C. G. Energy-efficient connectivity-aware learning over time-varying D2D networks. IEEE J. Sel. Top. Signal Process.(2024)

    2024

  35. [43]

    T., Langberg, M., Su, W

    Abdisarabshali, P., Kim, K. T., Langberg, M., Su, W. & Hosseinalipour, S. Dynamic D2D-assisted federated learning over O-RAN: Performance analysis, MAC scheduler, and asymmetric user selection. arXiv preprint arXiv:2404.06324 (2024)

    2024

  36. [44]

    & Brinton, C

    Zehtabi, S., Han, D.-J., Parasnis, R., Hosseinalipour, S. & Brinton, C. G. Decentralized sporadic federated learning: A unified methodology with generalized convergence guarantees. arXiv e-prints arXiv–2402 (2024)

    2024

  37. [45]

    P.-C., Hosseinalipour, S., Azam, S

    Lin, F. P.-C., Hosseinalipour, S., Azam, S. S., Brinton, C. G. & Michelusi, N. Semi-decentralized federated learning with cooperative D2D local model aggregations. IEEE J. on Sel. Areas Commun. 39, 3851–3869 (2021)

    2021

  38. [46]

    & Smith, V

    Kuo, K., Raje, A., Rajesh, K. & Smith, V . Federated LoRA with sparse communication.arXiv preprint arXiv:2406.05233 (2024)

    2024 arXiv

  39. [47]

    K., Prabhakar, G.et al

    Rajan, S. K., Prabhakar, G.et al. Heads up eXperience (HUX): Always-on AI companion for human computer environment interaction. arXiv preprint arXiv:2407.19492 (2024)

    2024 arXiv

  40. [48]

    & Cai, S

    Xu, J., He, S., Jiang, H., Yang, Y . & Cai, S. Design and implementation of an english lesson based on handwriting recognition and augmented reality in primary school. Int. Assoc. for Dev. Inf. Soc. (2019)

    2019

  41. [49]

    Raj, S. et al. Augmented reality and deep learning based system for assisting assembly process. J. on Multimodal User Interfaces 18, 119–133 (2024)

    2024

  42. [50]

    Musk, E. et al. An integrated brain-machine interface platform with thousands of channels. J. medical Internet research 21, e16194 (2019)

    2019

  43. [51]

    Y ., Hieu, N

    Zhu, H. Y ., Hieu, N. Q., Hoang, D. T., Nguyen, D. N. & Lin, C.-T. A human-centric metaverse enabled by brain-computer interface: A survey. IEEE Commun. Surv. & Tutorials (2024)

    2024

  44. [52]

    Effects of a brain-computer interface with virtual reality (VR) neurofeedback: A pilot study in chronic stroke patients

    V ourvopoulos, A.et al. Effects of a brain-computer interface with virtual reality (VR) neurofeedback: A pilot study in chronic stroke patients. Front. human neuroscience13, 210 (2019). 16/18

    2019

  45. [53]

    Nag, A. et al. TumorGANet: A transfer learning and generative adversarial network-based data augmentation model for brain tumor classification. IEEE Access (2024)

    2024

  46. [54]

    Efficacy and brain imaging correlates of an immersive motor imagery BCI-driven VR system for upper limb motor rehabilitation: A clinical case report

    V ourvopoulos, A.et al. Efficacy and brain imaging correlates of an immersive motor imagery BCI-driven VR system for upper limb motor rehabilitation: A clinical case report. Front. human neuroscience13, 244 (2019)

    2019

  47. [55]

    Yousri, R. et al. Illusionx: An LLM-powered mixed reality personal companion. arXiv preprint arXiv:2402.07924 (2024)

    2024 arXiv

  48. [56]

    & i Badia, S

    Lima, R., Chirico, A., Gaggioli, A., Gamboa, H. & i Badia, S. B. Real-time emotion regulation in virtual reality: An adaptive experience using breathing biofeedback. preprint (Version 1) available at Res. Sq.(2025)

    2025

  49. [57]

    & Collins, C

    Merrill Jr, K., Kim, J. & Collins, C. AI companions for lonely individuals and the role of social presence. Commun. Res. Reports 39, 93–103 (2022)

    2022

  50. [58]

    & Chen, W

    Eerdenisuyila, E., Li, H. & Chen, W. The analysis of generative adversarial network in sports education based on deep learning. Sci. Reports 14, 1–14 (2024)

    2024

  51. [59]

    Jafari, N. et al. Design strategies to foster improved experiences for patients in rehabilitation. HERD: Heal. Environ. Res. & Des. J. 19375867251346497 (2025)

    2025

  52. [60]

    Bovo, R. et al. Embardiment: an embodied AI agent for productivity in XR. In IEEE Conf. Virtual Reality 3D User Interfaces (VR), 708–717 (IEEE, 2025)

    2025

  53. [61]

    & Fonseca, M

    Lopes, P., Fachada, N. & Fonseca, M. Closing the loop: A systematic review of experience-driven game adaptation. arXiv preprint arXiv:2505.01351 (2025)

    2025 arXiv

  54. [62]

    U., Joy, A

    Kiafar, B., Ravva, P. U., Joy, A. A., Daher, S. & Barmaki, R. L. MENA: Multimodal epistemic network analysis for visualizing competencies and emotions. arXiv preprint arXiv:2504.02794 (2025)

    2025 arXiv

  55. [63]

    Savas ¸, E. H. et al. A biofeedback based virtual reality game for pediatric population (BioVirtualPed): A feasibility trial. In Seminars in Oncology Nursing, vol. 40, 151615 (Elsevier, 2024)

    2024

  56. [64]

    Klein Haneveld, L. et al. Exploring the added value of virtual reality biofeedback game deep in forensic psychiatric inpatient care—a qualitative study. Front. psychology 14, 1201485 (2023)

    2023

  57. [65]

    S., Silva, B

    de Lima, E. S., Silva, B. M. & Galam, G. T. Towards the design of adaptive virtual reality horror games: A model of players’ fears using machine learning and player modeling. In Brazilian Symp. Comput. Games Digital Entertainment (SBGames), 171–177 (IEEE, 2020)

    2020

  58. [66]

    Zaib, S. E. & Yamamura, M. Using heart rate and machine learning for VR horror game personalization. In 2022 IEEE Conference on Games (CoG), 213–220 (IEEE, 2022)

    2022

  59. [67]

    & Hosseinalipour, S

    Borazjani, K., Abdisarabshali, P., Khosravan, N. & Hosseinalipour, S. Redefining non-IID data in federated learning for computer vision tasks: Migrating from labels to embeddings for task-specific data distributions. arXiv preprint arXiv:2503.14553 (2025)

    2025

  60. [68]

    & Mahnan, A

    Di Luca, M. & Mahnan, A. Perceptual limits of visual-haptic simultaneity in virtual reality interactions. In 2019 IEEE World Haptics Conference (WHC), 67–72 (IEEE, 2019)

    2019

  61. [69]

    Hatami, M. et al. A survey of the real-time metaverse: Challenges and opportunities. Futur. Internet 16, 379 (2024)

    2024

  62. [70]

    & Nguyen, V

    Mahjourian, N. & Nguyen, V . Multimodal object detection using depth and image data for manufacturing parts.arXiv preprint arXiv:2411.09062 (2024)

    2024 arXiv

  63. [71]

    N., Fenty, N

    Yeganeh, L. N., Fenty, N. S., Chen, Y ., Simpson, A. & Hatami, M. The future of education: A multi-layered metaverse classroom model for immersive and inclusive learning. Futur. Internet 17, 63 (2025)

    2025

  64. [72]

    Kirkpatrick, J. et al. Overcoming catastrophic forgetting in neural networks. Proc. national academy sciences 114, 3521–3526 (2017)

    2017

  65. [73]

    & Sitaraman, R

    Wang, L., Singh, S., Chakareski, J., Hajiesmaili, M. & Sitaraman, R. K. BONES: Near-optimal neural-enhanced video streaming. In Proc. SIGMETRICS/Performance Int’l Conf. on Measure. Analysis Comput. Syst.(ACM/IFIP, Venice, Italy, 2024)

    2024

  66. [74]

    & Sitaraman, R

    Wang, L., Singh, S., Chakareski, J., Hajiesmaili, M. & Sitaraman, R. K. Bones: Near-optimal neural-enhanced video streaming. Proc. ACM Meas. Analysis Comput. Syst. 8, 1–28 (2024)

    2024

  67. [75]

    Vr/ar immersive communication: Caching, edge computing, and transmission trade-offs

    Chakareski, J. Vr/ar immersive communication: Caching, edge computing, and transmission trade-offs. In Proc. Workshop Virtual Reality Augment. Reality Netw., 36–41 (2017). 17/18

    2017

  68. [76]

    & Yuksel, M

    Chakareski, J., Khan, M. & Yuksel, M. Toward enabling next-generation societal virtual reality applications for virtual human teleportation: A novel future system concept and computation-communication-signal representation trade-offs. IEEE Signal Process. Mag. 39, 22–41 (2022)

    2022

  69. [77]

    & Badino, H

    Tome, D., Peluse, P., Agapito, L. & Badino, H. xR-EgoPose: Egocentric 3D human pose from an HMD camera. In Proc. IEEE/CVF Int. Conf. Comput. Vision, 7728–7738 (2019)

    2019

  70. [78]

    Hoi4d: A 4D egocentric dataset for category-level human-object interaction

    Liu, Y .et al. Hoi4d: A 4D egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 21013–21022 (2022)

    2022

  71. [79]

    Reimat, I. et al. CWIPC-SXR: Point cloud dynamic human dataset for social XR. In Proc. 12th ACM Multimedia Syst. Conf., 300–306 (2021)

    2021

  72. [80]

    G., Freitas, D

    Lohesara, F. G., Freitas, D. R., Guillemot, C., Eguiazarian, K. & Knorr, S. Headset: Human emotion awareness under partial occlusions multimodal dataset. IEEE Trans. Vis. Comput. Graph.29, 4686–4696 (2023)

    2023

  73. [81]

    Bhattacharya, A. et al. AMIS: An audiovisual dataset for multimodal XR research. In Proc. 16th ACM Multimedia Syst. Conf., 291–297 (2025)

    2025

  74. [82]

    & Alhebshi, R

    Alhoraibi, L., Alghazzawi, D. & Alhebshi, R. Generative adversarial network-based data augmentation for enhancing wireless physical layer authentication. Sensors 24, 641 (2024). Acknowledgements The authors acknowledge partial support from the National Science Foundation (NSF)...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.