REVIEW 4 major objections 3 minor 40 references
A missing-modality multimodal foundation model can learn a unified representation from RGB, depth, LiDAR, and CSI, and transfer that representation to UAV localization, beam prediction, and CSI prediction even when one sensing modality is a
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 04:27 UTC pith:TE5I7VTE
load-bearing objection Application-level novelty is real, but the missing dataset-split details and an undefined pooling operation make the reported numbers hard to trust as they stand. the 4 major comments →
M3F-UAV: A Missing-Modality Multimodal Foundation Model for Low-Altitude Wireless Sensing
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
M3F-UAV demonstrates that stochastic modality dropping, target-modality masking, and adaptive token pooling let a Transformer fusion encoder ingest variable modality subsets as fixed-length token sequences, so a single frozen or lightly fine-tuned backbone supports multiple downstream tasks. Pretraining jointly optimizes feature-level masked reconstruction of a randomly chosen target modality and UAV coordinate regression; this is what induces cross-modal correspondence and spatial awareness. On the LAMBDA urban subset, the fully fine-tuned model achieves the best average performance on all three tasks, and the missing-modality variants lose only modestly relative to the full-modality model,
What carries the argument
The load-bearing component is the multimodal token adaptation module: modality-specific pretrained encoders (MobileNetV2-style for RGB/depth, a Point Transformer for LiDAR, a Transformer for CSI) project their tokens into a shared dimension, then stochastic modality dropping, target-modality masking, and adaptive pooling convert the variable-length, partial token sequence into a fixed-size representation that a shared cross-modal Transformer encoder can fuse. This fixed-size property is what allows one backbone and one task head to serve every modality combination.
Load-bearing premise
The evaluation assumes the LAMBDA urban-subset train/test split is spatially and temporally disjoint, so the reported gains reflect generalization rather than near-memorization of adjacent frames; the paper does not specify the split or sample counts.
What would settle it
Re-run the same protocol with a train/test split that explicitly separates trajectories or temporally distant frames (e.g., train on scenarios 1-3 and test on scenario 4, or leave out contiguous time blocks) and check whether the reported margins over single-modality baselines persist. If localization error and next-step CSI NMSE jump to near-baseline levels, the headline results are artifacts of frame-level interpolation.
If this is right
- One pretrained backbone suffices for multiple low-altitude wireless tasks (localization, beam prediction, CSI prediction) with lightweight MLP heads, reducing per-task model maintenance.
- The model can be fine-tuned with as little as 10% of labeled data and still beat fully trained single-modality baselines on localization, suggesting data efficiency.
- Missing a modality causes only limited degradation in most settings; CSI is the most critical modality for beam prediction, while visual cues matter most for localization.
- Failure-aware gating can detect corrupted modalities and remove them before fusion, recovering most of the performance lost to sensor failure.
Where Pith is reading between the lines
- The design pattern generalizes beyond UAVs: any sensing stack with heterogeneous, intermittently available sensors (autonomous driving, IoT, robotics) could adopt the drop-and-reconstruct pretraining recipe.
- The paper's adaptive pooling discards token-position information across modalities; an inference is that tasks requiring fine-grained spatial alignment (e.g., precise angle-of-arrival estimation) might need a stronger positional or geometric prior than the current pooling provides.
- The evaluation is restricted to at most one missing modality; the paper claims support for arbitrary subsets during pretraining, so an editorial testable extension is to stress multi-missing combinations and see if fixed-size pooling still holds without performance collapse.
- CSI prediction is only one-step-ahead; extending to multi-step forecasting could reveal whether the learned temporal structure is truly predictive or mostly interpolative.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes M3F-UAV, a multimodal foundation model for low-altitude wireless sensing that fuses RGB images, depth maps, LiDAR point clouds, and CSI matrices. A shared Transformer-based fusion backbone is pretrained with stochastic modality dropping, feature-level masked reconstruction, and a UAV localization regression objective, and then adapted to downstream UAV localization, beam prediction, and CSI prediction via lightweight MLP heads. A Failure-Aware Modality Gating (FAMG) module is added to detect and discard corrupted modalities at inference. Experiments on the LAMBDA urban subset are reported for three downstream tasks, with missing-modality robustness, pretraining ablations, and FAMG comparisons showing that the full model usually outperforms single-modality baselines.
Significance. If the empirical claims hold, M3F-UAV would be a useful contribution to multimodal wireless sensing, particularly the missing-modality-robust fusion and the use of a public dataset for evaluation. The paper's strengths are its clear architectural decomposition, the inclusion of pretraining ablations, and the explicit attempt to address sensor failure. However, the evaluation relies on unverified data partitioning, the central adaptive pooling operator is not defined, and the localization downstream task overlaps with the pretraining objective. These issues are load-bearing for the headline claims, so the manuscript needs substantial revision to be credible.
major comments (4)
- [Section III.A, III.C, Tables IV, VI, VIII] The train/validation/test split is never specified. The manuscript states the urban subset contains four scenarios with certain weather/trajectory configurations (Table II) but does not state how frames are partitioned, whether partitions are temporally or spatially disjoint, the number of samples in each split, or the minimum temporal gap between train and test frames. At 60 Hz video, 20 Hz LiDAR, and a one-step CSI target (Eq. 67), a random frame split makes test samples near-duplicates of training samples, so localization, beam, and CSI metrics could largely reflect interpolation rather than generalization. This is the most load-bearing unverified condition; the paper should report results on a temporally and/or spatially disjoint split, state sample counts, and provide error bars or multiple split seeds.
- [Eq. (50), Section II.C.1] The adaptive token pooling operation ATP is never defined. It is the key mechanism claimed to convert variable-length multimodal token sequences into fixed-size representations, and the whole fusion design depends on it. The text only says 'adaptive token pooling' is applied to the concatenated multimodal sequence. Without a precise definition (learned vs. fixed, how the pooled token count K_a is selected, how mask tokens and missing modalities are handled, how variable token lengths across modalities are treated), the method is not reproducible from the manuscript.
- [Eqs. (59), (63); Section III.B.1] UAV localization is both a pretraining objective and a downstream evaluation task, and the same base-station position vector v_BS is fed into the modality-specific regression heads (Eqs. 16, 29, 41) and the fusion localization head (Eq. 57). Therefore the localization results demonstrate continuation of the pretraining task, not generalization to a new task. The localization tables (IV and V) should be either removed from the downstream claims, or the localization objective should be excluded from pretraining, or the paper should demonstrate transfer to held-out tasks not seen in pretraining. Otherwise the central 'foundation model' claim is weakened.
- [Eqs. (55)-(56), Section II.C.3] The feature-level reconstruction target \tilde{T}_t is not operationally defined. The manuscript says 'its unmasked adapted feature \tilde{T}_t is used as the reconstruction target', but it is not specified whether this target comes from the input adaptation path with stop-gradient, whether gradients flow through the target during pretraining, or how it relates to the masked token representation A^t. Without this, the latent reconstruction loss (Eq. 56) could admit trivial solutions or collapse. The output shape in Eq. (55) uses K_t while the adaptation produces K_a tokens; the relationship between K_t and K_a needs clarification.
minor comments (3)
- [Section III.D and II.E] Implementation details are incomplete: no optimizer, learning rate schedule, batch size, number of epochs, K_a value, FAMG training procedure, or failure-simulation noise levels are given. These are needed for reproducibility.
- [Table IV] The row for Scene 1 contains '0.367 0.3970.0440.060 0.048' with missing spaces, which makes the table difficult to read. Also, the text reports that W/o Rec has the best localization performance (e.g., average 0.092 vs. 0.121 for full M3F-UAV), so the claim that the complete pretraining strategy is beneficial for localization should be phrased more carefully.
- [References] References [1] and [32] appear to be the same work (6G WavesFM); please check for duplication or merge them.
Circularity Check
UAV localization is both a pretraining objective and a reported downstream transfer result, making that part of the evaluation partly forced; beam and CSI predictions remain independent.
specific steps
-
fitted input called prediction
[Sec. II.C.3 (Eqs. 57–59); Sec. III.B.1 (Eq. 63); Sec. III.C; Sec. III.E.1; Table IV]
"All modality-specific feature extractors are first pretrained on the available training data. The multimodal fusion model is then pretrained on the same data. ... The second pretraining task is UAV coordinate regression. ... L_loc = ||ŷ−y||^2_2, where y ∈ R^3 is the ground-truth UAV position. ... W/o Rec achieves the best localization performance, which is reasonable because it is pretrained only with the regression objective and is therefore more aligned with the downstream localization task."
Table IV reports UAV localization as a downstream transfer result, but UAV localization is one of the two pretraining objectives (Eq. 59), optimized on the same training labels before fine-tuning. The downstream head (Eq. 63) predicts the same 3-D coordinate y from the same pooled fusion feature f that Eq. 58 already regressed during pretraining. The paper's own ablation comment confirms that localization performance tracks the pretraining objective. Thus the reported localization gains—and especially the '10% fine-tuning' data-efficiency claim—are partly a continuation of the pretraining fit rather than an independent transfer test.
full rationale
The core architecture and the other two downstream tasks are not circular: feature-level masked reconstruction (Eq. 56) is a standard self-supervised objective, and beam prediction (Table VI) and CSI prediction (Table VIII) are not pretraining objectives. No load-bearing self-citation chain or imported uniqueness theorem is used; LAMBDA is an external dataset. The circularity is confined to the UAV localization evaluation: because localization is explicitly a pretraining objective with the same labels and the same fused feature, the localization numbers are partially forced by construction. The unspecified LAMBDA train/test split is a serious correctness risk but not a circularity reduction, since the paper does not define the split in terms of its outputs. Overall, partial circularity in one of three headline evaluations justifies a score of 5.
Axiom & Free-Parameter Ledger
free parameters (6)
- adaptive pooling output length K_a =
not reported
- masking ratio =
50%
- fusion encoder depth/width =
L=4, heads=4, D=256
- FPS patch count G and KNN K =
not reported
- failure simulation noise levels =
not reported
- optimizer/batch size/learning rates =
not reported
axioms (4)
- domain assumption LAMBDA provides accurate ground-truth UAV positions and beam labels
- domain assumption 4.9 GHz CSI carries spatial information sufficient to predict 28 GHz beams
- domain assumption The urban subset split prevents temporal leakage
- ad hoc to paper The learnable projection output eTt is a valid reconstruction target without stop-gradient
read the original abstract
Low-altitude unmanned aerial vehicles (UAVs) are emerging as key platforms for wireless intelligence tasks. However, practical low-altitude wireless systems usually operate in complex urban environments, where visual occlusion, sparse geometric observations, multipath propagation, and sensor failures may degrade the reliability of single-modality models. To address these challenges, this paper proposes M3F-UAV, a missing-modality multimodal foundation model for low-altitude wireless sensing. The proposed framework learns a unified multimodal representation from visual, geometric, and wireless observations. Specifically, modality-specific pretrained feature extractors are adopted for RGB/depth images, LiDAR point clouds, and CSI matrices, respectively. Through cross-modal fusion and missing-modality-aware pretraining with feature-level masked reconstruction and UAV localization objectives, M3F-UAV can extract fixed-size features from different modality combinations and adapt them to downstream low-altitude wireless tasks with lightweight task heads. Experiments on the LAMBDA dataset show that M3F-UAV outperforms single-modality baselines and maintains robust performance under missing-modality settings.
Figures
Reference graph
Works this paper leans on
-
[1]
6g wavesfm: A foundation model for sensing, communication, and localization,
A. AboElfotouh, E. Mohammed, and H. Abou-Zeid, “6g wavesfm: A foundation model for sensing, communication, and localization,”IEEE Open J. Commun. Soc., vol. 6, pp. 6792–6807, 2025
2025
-
[2]
Survey of important issues in UA V communication networks,
L. Gupta, R. Jain, and G. Vaszkun, “Survey of important issues in UA V communication networks,”IEEE Commun. Surv. Tutorials, vol. 18, no. 2, pp. 1123–1152, 2016
2016
-
[3]
A survey of sensors based autonomous unmanned aerial vehicle UA V localization techniques,
H. Liu, Q. Long, B. Yiet al., “A survey of sensors based autonomous unmanned aerial vehicle UA V localization techniques,”Complex Intell. Syst., vol. 11, no. 8, p. 371, 2025
2025
-
[4]
Low-altitude wireless networks: A comprehensive survey,
J. Wu, Y . Yang, W. Yuanet al., “Low-altitude wireless networks: A comprehensive survey,” 2026, arXiv:2509.11607
Pith/arXiv arXiv 2026
-
[5]
A survey of channel modeling for UA V communications,
A. A. Khuwaja, Y . Chen, N. Zhaoet al., “A survey of channel modeling for UA V communications,”IEEE Commun. Surv. Tutorials, vol. 20, no. 4, pp. 2804–2821, 2018. 13
2018
-
[6]
UA V detection and tracking in urban en- vironments using passive sensors: A survey,
X. Yan, T. Fu, H. Linet al., “UA V detection and tracking in urban en- vironments using passive sensors: A survey,”Applied Sciences, vol. 13, no. 20, p. 11320, 2023
2023
-
[7]
UA V position estimation using a LiDAR- based 3d object detection method,
U. Olawoye and J. N. Gross, “UA V position estimation using a LiDAR- based 3d object detection method,”CoRR, vol. abs/2504.07028, 2025
Pith/arXiv arXiv 2025
-
[8]
Localization of unmanned aerial vehicles in corridor environments using deep learning,
R. P. Padhy, S. Ahmad, S. Vermaet al., “Localization of unmanned aerial vehicles in corridor environments using deep learning,” 2019
2019
-
[9]
CSI-based fingerprinting for indoor localization: A deep learning approach,
X. Wang, L. Gao, S. Maoet al., “CSI-based fingerprinting for indoor localization: A deep learning approach,” 2016
2016
-
[10]
Towards real-world 6G drone communication: Position and camera aided beam prediction,
G. Charan, A. Hredzak, C. Stoddardet al., “Towards real-world 6G drone communication: Position and camera aided beam prediction,” in Proc. IEEE Global Commun. Conf., 2022, pp. 2951–2956
2022
-
[11]
Machine learning on camera images for fast mmwave beamforming,
B. Salehi, M. Belgiovine, S. G. Sanchezet al., “Machine learning on camera images for fast mmwave beamforming,” inMASS, 2020, pp. 338–346
2020
-
[12]
LiDAR aided future beam prediction in real-world millimeter wave V2I communications,
S. Jiang, G. Charan, and A. Alkhateeb, “LiDAR aided future beam prediction in real-world millimeter wave V2I communications,” 2022, arXiv:2203.05548
Pith/arXiv arXiv 2022
-
[13]
Radar aided 6G beam prediction: Deep learning algorithms and real-world demonstration,
U. Demirhan and A. Alkhateeb, “Radar aided 6G beam prediction: Deep learning algorithms and real-world demonstration,” inProc. IEEE Wireless Commun. Netw. Conf., 2022, pp. 2655–2660
2022
-
[14]
Deep learning-based channel prediction for TDD MIMO systems with imperfect channel reciprocity,
H. Yang, “Deep learning-based channel prediction for TDD MIMO systems with imperfect channel reciprocity,”ICT Express, vol. 11, no. 3, pp. 590–596, 2025
2025
-
[15]
Long short-term memory network for wireless channel prediction,
X. Tong and S. Sun, “Long short-term memory network for wireless channel prediction,” inProc. Int. Conf. Signal Inf. Process., Netw. Comput., 2018, pp. 19–26
2018
-
[16]
Deepsense 6G: A large- scale real-world multi-modal sensing and communication dataset,
A. Alkhateeb, G. Charan, T. Osmanet al., “Deepsense 6G: A large- scale real-world multi-modal sensing and communication dataset,”IEEE Commun. Mag., vol. 61, no. 9, pp. 122–128, 2023
2023
-
[17]
Vision-position multi-modal beam prediction using real millimeter wave datasets,
G. Charan, T. Osman, A. Hredzaket al., “Vision-position multi-modal beam prediction using real millimeter wave datasets,” inProc. IEEE Wireless Commun. Netw. Conf., 2022, pp. 2727–2731
2022
-
[18]
Vision-assisted beam prediction for real world 6G drone communication,
I. Ahmad, A. R. Khan, R. N. B. Raiset al., “Vision-assisted beam prediction for real world 6G drone communication,” inProc. IEEE 34th Annu. Int. Symp. Pers., Indoor Mobile Radio Commun., 2023, pp. 1–7
2023
-
[19]
Multimodal trans- formers for wireless communications: A case study in beam prediction,
Y . Tian, Q. Zhao, Z. el abidine Kherroubiet al., “Multimodal trans- formers for wireless communications: A case study in beam prediction,” 2023
2023
-
[20]
Multi-modal sensing-assisted beam prediction using real-world dataset,
Y . Yeo, J. Kim, J. Kimet al., “Multi-modal sensing-assisted beam prediction using real-world dataset,”J. Commun. Networks, vol. 27, no. 5, pp. 412–419, 2025
2025
-
[21]
M2BeamLLM: Multimodal sensing- empowered mmwave beam prediction with large language models,
C. Zheng, J. He, C. G. Kanget al., “M2BeamLLM: Multimodal sensing- empowered mmwave beam prediction with large language models,” 2025, arXiv:2506.14532
Pith/arXiv arXiv 2025
-
[22]
A novel multimodal fusion sensing-based channel prediction method for uav communications,
Z. Xin, Y . Liu, J. Xinget al., “A novel multimodal fusion sensing-based channel prediction method for uav communications,”IEEE Internet Things J., vol. 12, no. 4, pp. 3948–3960, 2025
2025
-
[23]
Language models are few-shot learners,
T. B. Brown, B. Mann, N. Ryderet al., “Language models are few-shot learners,” inProc. Adv. Neural Inf. Process. Syst., 2020
2020
-
[24]
PaLM: Scaling language modeling with pathways,
A. Chowdhery, S. Narang, J. Devlinet al., “PaLM: Scaling language modeling with pathways,”J. Mach. Learn. Res., vol. 24, pp. 1–113, 2023
2023
-
[25]
LLaMA: Open and efficient foundation language models,
H. Touvron, T. Lavril, G. Izacardet al., “LLaMA: Open and efficient foundation language models,” 2023, arXiv:2302.13971
Pith/arXiv arXiv 2023
-
[26]
TelecomGPT: A framework to build telecom-specific large language models,
H. Zou, Q. Zhao, Y . Tianet al., “TelecomGPT: A framework to build telecom-specific large language models,”IEEE Trans. Mach. Learn. Commun. Netw., vol. 3, pp. 948–975, 2025
2025
-
[27]
LLM4CP: adapting large language models for channel prediction,
B. Liu, X. Liu, S. Gaoet al., “LLM4CP: adapting large language models for channel prediction,”J. Commun. Inf. Networks, vol. 9, no. 2, pp. 113–125, 2024
2024
-
[28]
LLM4WM: Adapting llm for wireless multi-tasking,
X. Liu, S. Gao, B. Liuet al., “LLM4WM: Adapting llm for wireless multi-tasking,”IEEE Trans. Mach. Learn. Commun. Netw., vol. 3, pp. 835–847, 2025
2025
-
[29]
Large Wireless Model (LWM): A foundation model for wireless channels,
S. Alikhani, G. Charan, and A. Alkhateeb, “Large Wireless Model (LWM): A foundation model for wireless channels,” 2025, arXiv:2411.08872
Pith/arXiv arXiv 2025
-
[30]
WiFo: wireless foundation model for channel prediction,
B. Liu, S. Gao, X. Liuet al., “WiFo: wireless foundation model for channel prediction,”Sci. China Inf. Sci., vol. 68, no. 6, 2025
2025
-
[31]
AirFM-DDA: Air-interface foundation model in the delay-doppler-angle domain for AI-native 6G,
K. Bian, M. Tao, J. Moet al., “AirFM-DDA: Air-interface foundation model in the delay-doppler-angle domain for AI-native 6G,” 2026
2026
-
[32]
6G WavesFM: A foundation model for sensing, communication, and localization,
A. AboElfotouh, E. Mohammed, and H. Abou-Zeid, “6G WavesFM: A foundation model for sensing, communication, and localization,”IEEE Open J. Commun. Soc., vol. 6, pp. 6792–6807, 2025
2025
-
[33]
A MIMO wireless channel foundation model via CIR-CSI consistency,
J. Jiang, W. Yu, Y . Liet al., “A MIMO wireless channel foundation model via CIR-CSI consistency,” 2025
2025
-
[34]
MUSE-FM: Multi-task environment- aware foundation model for wireless communications,
T. Zheng, J. Guo, L. Daiet al., “MUSE-FM: Multi-task environment- aware foundation model for wireless communications,” 2026
2026
-
[35]
Multimodal mixture-of-experts for ISAC in low-altitude wireless networks,
K. Zhang, W. Yu, H. Heet al., “Multimodal mixture-of-experts for ISAC in low-altitude wireless networks,” 2025
2025
-
[36]
MultiMAE: Multi-modal multi-task masked autoencoders,
R. Bachmann, D. Mizrahi, A. Atanovet al., “MultiMAE: Multi-modal multi-task masked autoencoders,” inProc. Eur. Conf. Comput. Vis., 2022, pp. 348–367
2022
-
[37]
Towards robust multimodal physio- logical foundation models: Handling arbitrary missing modalities,
W.-B. Jiang, X. Fu, Y . Dinget al., “Towards robust multimodal physio- logical foundation models: Handling arbitrary missing modalities,” 2026, arXiv:2504.19596
arXiv 2026
-
[38]
X-Fi: A modality-invariant foundation model for multimodal human sensing,
X. Chen and J. Yang, “X-Fi: A modality-invariant foundation model for multimodal human sensing,” inProc. 13th Int. Conf. Learn. Represent., 2025
2025
-
[39]
MobileNetV2: Inverted residuals and linear bottlenecks,
M. Sandler, A. G. Howard, M. Zhuet al., “MobileNetV2: Inverted residuals and linear bottlenecks,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 4510–4520
2018
-
[40]
Masked autoencoders for point cloud self-supervised learning,
Y . Pang, W. Wang, F. E. H. Tayet al., “Masked autoencoders for point cloud self-supervised learning,” inProc. Eur. Conf. Comput. Vis., 2022, pp. 604–621
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.