Resolving Multi-Target Association in OFDM-based ISAC via Vision-aided Multi-Modal Learning
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-26 12:04 UTCgrok-4.3pith:CC2KWCIZrecord.jsonopen to challenge →
The pith
Fusing reconstructed street-view images with delay-Doppler maps resolves target association ambiguities in multi-target OFDM-ISAC.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The vision-assisted OFDM-ISAC framework resolves both data-association ambiguity and resolution limits by encoding a street-view image with DeepJSCC over the sensing waveform, reconstructing it at the receiver, extracting target features with a fine-tuned YOLOv5 detector, and fusing those features with the delay-Doppler map and transmitter-receiver geometry in a multi-modal network, yielding 16 cm localization RMSE and 10.8 ns delay RMSE on a Blender-rendered vehicular testbed.
What carries the argument
The multi-modal fusion network that combines YOLOv5 bounding-box coordinates and class labels from the reconstructed image with delay-Doppler map peaks and geometry information.
If this is right
- The visual modality supplies the missing association information that pure wireless peak-searching cannot provide.
- Kullback-Leibler loss against triangular soft labels stabilizes training of the high-dimensional delay and Doppler classifiers.
- Removing the visual modality produces a 60-fold increase in localization RMSE.
- The same OFDM waveform simultaneously carries the encoded image and the sensing pilots.
Where Pith is reading between the lines
- The method could extend to cooperative perception if multiple vehicles share reconstructed views over the same waveform.
- Performance in real-world weather or lighting changes would depend on how well DeepJSCC and YOLOv5 generalize beyond the rendered scenes.
- Similar fusion of visual features might reduce pilot overhead in other ISAC settings such as indoor radar or drone tracking.
Load-bearing premise
The Blender-rendered vehicular testbed and the assumption that image reconstruction and YOLOv5 detection remain reliable enough under the same channel conditions used for sensing to provide unambiguous target association in all tested scenes.
What would settle it
An experiment in which the reconstructed image fails to detect or correctly label at least one target that produces a distinct peak in the delay-Doppler map, resulting in localization RMSE substantially larger than 16 cm.
Figures
read the original abstract
Orthogonal frequency division multiplexing (OFDM)-based integrated sensing and communication (ISAC) systems commonly extract target parameters by peak-searching a delay-Doppler map (DDM) constructed from reflected pilots. In multi-target scenarios, this results in ambiguity: the DDM does not reveal which physical target produced which peak, and two targets within the same delay-Doppler resolution cell cannot be separated. We propose a vision-assisted OFDM-ISAC framework that resolves both limitations by fusing wireless and visual modalities. The transmitter encodes an onboard street-view image with deep joint source-channel coding (DeepJSCC) and transmits it over the same OFDM waveform used for sensing; the receiver reconstructs the image, runs a fine-tuned YOLOv5 detector and fuses the resulting per-target features (bounding-box coordinates and class labels) with the DDM and transmitter-receiver geometry through a learned multi-modal network. To stabilize training of the high dimensional delay and Doppler classifiers, we introduce a Kullback Leibler loss against triangular soft labels centered on the ground-truth bin. On a Blender-rendered vehicular testbed, the proposed framework achieves a 16 cm localization root mean square error (RMSE) and a 10.8 ns delay RMSE. An ablation study confirms that removing the visual modality causes a 60x degradation in localization. These results highlight the potential of vision to overcome the data-association and resolution limits of single-modality ISAC.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a vision-assisted multi-modal framework for OFDM-ISAC that transmits DeepJSCC-encoded street-view images over the sensing waveform, reconstructs them at the receiver, applies fine-tuned YOLOv5 detection, and fuses per-target bounding boxes and labels with DDM peaks plus geometry in a learned network. A KL loss against triangular soft labels is used to train high-dimensional delay/Doppler classifiers. On a Blender-rendered vehicular testbed the framework reports 16 cm localization RMSE and 10.8 ns delay RMSE, with an ablation showing 60× degradation when the visual modality is removed.
Significance. If the quantitative results hold, the work supplies concrete empirical evidence that visual fusion can resolve data-association ambiguity and improve resolution beyond single-modality DDM peak search in simulated ISAC settings. The large ablation effect size and the specific RMSE numbers, together with the DeepJSCC + YOLO integration, constitute a clear contribution to multi-modal sensing research.
major comments (2)
- [Simulation setup and ablation study] The Blender testbed generates visual and wireless observations from the identical virtual scene, so target association and modality correlation hold by construction. This directly supports the reported 60× localization gain but leaves open whether the multi-modal network would maintain performance under realistic mismatches (camera-radar calibration drift, lighting variation, or independent reconstruction errors). The central performance claims therefore rest on an idealized correlation that the manuscript does not stress-test.
- [Results section] The abstract and results present concrete RMSE figures (16 cm localization, 10.8 ns delay) and the 60× ablation factor without reporting the number of independent trials, random seeds, standard deviations, or whether simulation parameters were selected after inspecting test-set performance. These omissions make the quantitative claims difficult to interpret or reproduce.
minor comments (2)
- [Multi-modal fusion network] The description of how bounding-box coordinates and class labels are encoded as network inputs would benefit from an explicit equation or block diagram.
- [Introduction] A few sentences on related multi-modal ISAC or DeepJSCC-for-sensing literature would help situate the contribution.
Simulated Author's Rebuttal
We thank the referee for the constructive comments. We address each major comment below and indicate planned revisions to the manuscript.
read point-by-point responses
-
Referee: [Simulation setup and ablation study] The Blender testbed generates visual and wireless observations from the identical virtual scene, so target association and modality correlation hold by construction. This directly supports the reported 60× localization gain but leaves open whether the multi-modal network would maintain performance under realistic mismatches (camera-radar calibration drift, lighting variation, or independent reconstruction errors). The central performance claims therefore rest on an idealized correlation that the manuscript does not stress-test.
Authors: We agree that the simulation generates both modalities from the same virtual scene, creating idealized correlation by construction. This controlled setup isolates the contribution of multi-modal fusion to resolving association and resolution issues. We will revise the manuscript to add an explicit limitations subsection in the discussion that acknowledges this idealization and outlines potential effects of real-world mismatches such as calibration drift or lighting variation, together with directions for future robustness experiments. revision: yes
-
Referee: [Results section] The abstract and results present concrete RMSE figures (16 cm localization, 10.8 ns delay) and the 60× ablation factor without reporting the number of independent trials, random seeds, standard deviations, or whether simulation parameters were selected after inspecting test-set performance. These omissions make the quantitative claims difficult to interpret or reproduce.
Authors: We acknowledge that the manuscript omits these reproducibility details. We will revise the results section (and update the abstract if space permits) to report the number of independent trials, the random seeds used, the standard deviations of the reported RMSE metrics, and to confirm that simulation parameters were selected using only training and validation data. revision: yes
Circularity Check
No circularity; results are empirical outcomes from simulation testbed
full rationale
The paper presents an empirical multi-modal framework evaluated on a Blender-rendered vehicular testbed. Performance claims (16 cm localization RMSE, 10.8 ns delay RMSE, 60x ablation degradation) are reported as measured results after training the learned network on generated data pairs. No equations, derivations, or self-citations are shown that reduce any central claim to fitted inputs by construction. The KL loss and fusion network are standard training techniques whose outputs are validated externally on held-out simulation scenes rather than being tautological. The simulation alignment noted in the skeptic attack is an assumption about testbed fidelity, not a circular reduction in the derivation chain itself.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
A survey on fundamental limits of integrated sensing and communication,
A. Liu, Z. Huang, M. Li, Y . Wan, W. Li, T. X. Han, C. Liu, R. Du, D. K. P. Tan, J. Lu, Y . Shen, F. Colone, and K. Chetty, “A survey on fundamental limits of integrated sensing and communication,”IEEE Commun. Surveys Tuts., vol. 24, no. 2, pp. 994–1034, 2nd Quat. 2022
2022
-
[2]
The integrated sensing and communication revolution for 6G: Vision, techniques, and applications,
N. Gonz ´alez-Prelcic, M. Furkan Keskin, O. Kaltiokallio, M. Valkama, D. Dardari, X. Shen, Y . Shen, M. Bayraktar, and H. Wymeersch, “The integrated sensing and communication revolution for 6G: Vision, techniques, and applications,”Proc. IEEE, vol. 112, no. 7, pp. 676–723, Jul. 2024
2024
-
[3]
Optimal transmit beamforming for integrated sensing and communication,
H. Hua, J. Xu, and T. X. Han, “Optimal transmit beamforming for integrated sensing and communication,”IEEE Trans. V eh. Technol., vol. 72, no. 8, pp. 10 588–10 603, Aug. 2023
2023
-
[4]
Joint active and passive beamforming design for IRS-aided radar-communication,
M. Hua, Q. Wu, C. He, S. Ma, and W. Chen, “Joint active and passive beamforming design for IRS-aided radar-communication,”IEEE Trans. Wireless Commun., vol. 22, no. 4, pp. 2278–2294, Apr. 2023
2023
-
[5]
Joint beamforming and scheduling for integrated sensing and communication systems in URLLC: A POMDP approach,
X. Zhao and Y .-J. Angela Zhang, “Joint beamforming and scheduling for integrated sensing and communication systems in URLLC: A POMDP approach,”IEEE Trans. Commun., vol. 72, no. 10, pp. 6145–6161, Oct. 2024
2024
-
[7]
V ortex wavefront FMCW ISAC model: A blender-based evaluation,
Y . Liu, W.-X. Long, R. Chen, L. Wu, and M. R. B. Shankar, “V ortex wavefront FMCW ISAC model: A blender-based evaluation,” inIEEE 24th International Workshop on Signal Processing Advances in Wireless Communications (SPA WC), Shanghai, China, 2023, pp. 431–435
2023
-
[8]
An OFDM-ISAC scheme based on zadoff-Chu sequence for 5G NR base-station,
M. Wang, K. Yoshii, S. Shimamoto, M. Funakoshi, A. Yabuki, and H. Funayoshi, “An OFDM-ISAC scheme based on zadoff-Chu sequence for 5G NR base-station,” inIEEE 22nd Consumer Communications & Networking Conference (CCNC), Las Vegas, NV , USA, 2025, pp. 1–4
2025
-
[9]
Integrated sensing and communication: Joint pilot and transmission design,
M. Hua, Q. Wu, W. Chen, A. Jamalipour, C. Wu, and O. A. Dobre, “Integrated sensing and communication: Joint pilot and transmission design,”IEEE Trans. Wireless Commun., vol. 23, no. 11, pp. 16 017– 16 032, Nov. 2024
2024
-
[10]
Low- complexity minimum ber precoder design for ISAC systems: A delay- doppler perspective,
J. Wu, W. Yuan, Z. Wei, K. Zhang, F. Liu, and D. Wing Kwan Ng, “Low- complexity minimum ber precoder design for ISAC systems: A delay- doppler perspective,”IEEE Trans. Wireless Commun., vol. 24, no. 2, pp. 1526–1540, Feb. 2025
2025
-
[11]
Sensing aided uplink transmission in OTFS ISAC with joint parameter association, channel estimation and signal detection,
X. Yang, H. Li, Q. Guo, J. A. Zhang, X. Huang, and Z. Cheng, “Sensing aided uplink transmission in OTFS ISAC with joint parameter association, channel estimation and signal detection,”IEEE Trans. V eh. Technol., vol. 73, no. 6, pp. 9109–9114, 2024
2024
-
[12]
On the fundamental tradeoff of integrated sensing and communications under gaussian channels,
Y . Xiong, F. Liu, Y . Cui, W. Yuan, T. X. Han, and G. Caire, “On the fundamental tradeoff of integrated sensing and communications under gaussian channels,”IEEE Trans. Inf. Theory, vol. 69, no. 9, pp. 5723– 5751, Sept. 2023
2023
-
[13]
Fundamental CRB-rate tradeoff in multi-antenna isac systems with information multicasting and multi-target sensing,
Z. Ren, Y . Peng, X. Song, Y . Fang, L. Qiu, L. Liu, D. W. K. Ng, and J. Xu, “Fundamental CRB-rate tradeoff in multi-antenna isac systems with information multicasting and multi-target sensing,”IEEE Trans. Wireless Commun., vol. 23, no. 4, pp. 3870–3885, Apr. 2024
2024
-
[14]
Unified ISAC pareto boundary based on mutual information and minimum mean- square error estimation,
S. Wang, L. Chen, J. Zhou, Y . Chen, K. Han, and C. You, “Unified ISAC pareto boundary based on mutual information and minimum mean- square error estimation,”IEEE Trans. Commun., vol. 72, no. 11, pp. 6783–6795, Nov. 2024
2024
-
[15]
Cooperative ISAC networks: Opportunities and challenges,
K. Meng, C. Masouros, A. P. Petropulu, and L. Hanzo, “Cooperative ISAC networks: Opportunities and challenges,”IEEE Wireless Commun., vol. 32, no. 3, pp. 212–219, Jun. 2025
2025
-
[16]
Network-level performance analysis for air-ground integrated sensing and communication,
Y . Jiang, X. Li, G. Zhu, K. Han, K. Meng, W. Yang, C. Liu, Q. Shi, and R. Zhang, “Network-level performance analysis for air-ground integrated sensing and communication,”IEEE Trans. Wireless Commun., vol. 24, no. 8, pp. 6931–6946, Aug. 2025
2025
-
[17]
Network-level isac: An analytical study of antenna topologies ranging from massive to cell-free MIMO,
K. Meng, K. Han, C. Masouros, and L. Hanzo, “Network-level isac: An analytical study of antenna topologies ranging from massive to cell-free MIMO,”IEEE Trans. Wireless Commun., vol. 24, no. 12, pp. 10 003– 10 018, Dec. 2025
2025
-
[18]
Deep-learning-based techniques for integrated sensing and communication systems: State-of-the-art, challenges, and opportu- nities,
M. Temiz, Y . Zhang, Y . Fu, C. Zhang, C. Meng, O. Kaplan, and C. Masouros, “Deep-learning-based techniques for integrated sensing and communication systems: State-of-the-art, challenges, and opportu- nities,”IEEE Open J. Commun. Soc., vol. 6, pp. 5940–5968, Jul. 2025
2025
-
[19]
ISAC- NET: Model-driven deep learning for integrated passive sensing and communication,
W. Jiang, D. Ma, Z. Wei, Z. Feng, P. Zhang, and J. Peng, “ISAC- NET: Model-driven deep learning for integrated passive sensing and communication,”IEEE Trans. Commun., vol. 72, no. 8, pp. 4692–4707, 2024
2024
-
[20]
Deep-learning-based multinode ISAC 4D environmental reconstruction with uplink–downlink cooperation,
B. Lu, Z. Wei, H. Wu, X. Zeng, L. Wang, X. Lu, D. Mei, and Z. Feng, “Deep-learning-based multinode ISAC 4D environmental reconstruction with uplink–downlink cooperation,”IEEE Internet Things J., vol. 11, no. 24, pp. 39 512–39 526, Dec. 2024
2024
-
[21]
Vision-aided 6G wireless communications: Blockage prediction and proactive handoff,
G. Charan, M. Alrabeiah, and A. Alkhateeb, “Vision-aided 6G wireless communications: Blockage prediction and proactive handoff,”IEEE Trans. V eh. Technol., vol. 70, no. 10, pp. 10 193–10 208, 2021
2021
-
[22]
Vision-aided ultra- reliable low-latency communications for smart factory,
Y . Feng, F. Gao, X. Tao, S. Ma, and H. V . Poor, “Vision-aided ultra- reliable low-latency communications for smart factory,”IEEE Trans. Commun., vol. 72, no. 6, pp. 3439–3453, Jun. 2024
2024
-
[23]
Computer vision-based link scheduling in mmwave multi-hop V2X communica- tions,
W. Xu, C. Zhao, F. Gao, L. Xing, J. Qian, and H. Wang, “Computer vision-based link scheduling in mmwave multi-hop V2X communica- tions,”IEEE Trans. Commun., vol. 73, no. 8, pp. 5621–5634, Aug. 2025
2025
-
[24]
Vision image aided near-field beam training for internet of vehicle systems,
W. Huang, X. Huang, H. Zhang, K. Sun, C. Kai, and S. He, “Vision image aided near-field beam training for internet of vehicle systems,” inIEEE International Conference on Communications Workshops (ICC Workshops), Denver, CO, USA,2024, pp. 390–395
2024
-
[25]
Computer vision aided mmwave beam alignment in V2X communications,
W. Xu, F. Gao, X. Tao, J. Zhang, and A. Alkhateeb, “Computer vision aided mmwave beam alignment in V2X communications,”IEEE Trans. Wireless Commun., vol. 22, no. 4, pp. 2699–2714, Apr. 2023
2023
-
[26]
Multi-camera views based beam searching and BS selection with reduced training overhead,
B. Lin, F. Gao, Y . Zhang, C. Pan, and G. Liu, “Multi-camera views based beam searching and BS selection with reduced training overhead,”IEEE Trans. Commun., vol. 72, no. 5, pp. 2793–2805, May 2024
2024
-
[27]
Vision aided environment semantics extraction and its application in mmwave beam selection,
F. Wen, W. Xu, F. Gao, C. Pan, and G. Liu, “Vision aided environment semantics extraction and its application in mmwave beam selection,” IEEE Commun. Lett., vol. 27, no. 7, pp. 1894–1898, Jul. 2023
2023
-
[28]
Computer vision aided beam tracking in a real-world millimeter wave deployment,
S. Jiang and A. Alkhateeb, “Computer vision aided beam tracking in a real-world millimeter wave deployment,” inIEEE Globecom Workshops (GC Wkshps), Rio de Janeiro, Brazil, 2022, pp. 142–147
2022
-
[29]
Vision aided beam tracking and fre- quency handoff for mmwave communications,
T. Zhang, J. Liu, and F. Gao, “Vision aided beam tracking and fre- quency handoff for mmwave communications,” inIEEE Conference on Computer Communications Workshops (INFOCOM), 2022, pp. 1–2
2022
-
[30]
Semantic-aware vision-assisted integrated sensing and communication: Architecture and resource allocation,
Y . Lu, W. Mao, H. Du, O. A. Dobre, D. Niyato, and Z. Ding, “Semantic-aware vision-assisted integrated sensing and communication: Architecture and resource allocation,”IEEE Wireless Commun., vol. 31, no. 3, pp. 302–308, Jun. 2024
2024
-
[31]
LIDAR and position-aided mmwave beam selection with non-local CNNs and curriculum training,
M. Zecchin, M. B. Mashhadi, M. Jankowski, D. G ¨und¨uz, M. Kountouris, and D. Gesbert, “LIDAR and position-aided mmwave beam selection with non-local CNNs and curriculum training,”IEEE Trans. V eh. Tech- nol., vol. 71, no. 3, pp. 2979–2990, Mar. 2022
2022
-
[32]
Vision-position multi-modal beam prediction using real millimeter wave datasets,
G. Charan, T. Osman, A. Hredzak, N. Thawdar, and A. Alkhateeb, “Vision-position multi-modal beam prediction using real millimeter wave datasets,” inIEEE Wireless Communications and Networking Conference (WCNC), Austin, TX, USA, 2022, pp. 2727–2731
2022
-
[33]
Joint sensing and task-oriented communications with image and wireless data modali- ties for dynamic spectrum access,
Y . E. Sagduyu, T. Erpek, A. Yener, and S. Ulukus, “Joint sensing and task-oriented communications with image and wireless data modali- ties for dynamic spectrum access,” inIEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), Washington, DC, USA,2024, pp. 57–62
2024
-
[34]
Multi-modal beam prediction challenge 2022: Towards generalization,
G. Charan, U. Demirhan, J. Morais, A. Behboodi, H. Pezeshki, and A. Alkhateeb, “Multi-modal beam prediction challenge 2022: Towards generalization,”arXiv preprint arXiv:2209.07519, 2022
-
[35]
Vision-aided ISAC in low-altitude economy networks via de-diffused visual priors,
Y . Gao, Z. Ye, Z. Lyu, M. Xiao, Y . Xiao, P. Yang, and A. Manolova, “Vision-aided ISAC in low-altitude economy networks via de-diffused visual priors,”IEEE Trans. Cogn. Commun. Netw., vol. 12, pp. 3831– 3845, 2026
2026
-
[36]
Multimodal deep learning-empowered beam prediction in future THz ISAC systems,
K. Zhang, W. Yu, H. He, S. Song, J. Zhang, and K. B. Letaief, “Multimodal deep learning-empowered beam prediction in future THz ISAC systems,” inIEEE 36th International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), 2025
2025
-
[37]
MIMO-OFDM ISAC waveform design for range-Doppler sidelobe suppression,
P. Li, M. Li, R. Liu, Q. Liu, and A. Lee Swindlehurst, “MIMO-OFDM ISAC waveform design for range-Doppler sidelobe suppression,”IEEE Trans. Wireless Commun., vol. 24, no. 2, pp. 1001–1015, Feb. 2025
2025
-
[38]
3D multi-target localization via intelligent reflecting surface: Protocol and analysis,
M. Hua, G. Chen, K. Meng, S. Ma, C. Yuen, and H. Cheung So, “3D multi-target localization via intelligent reflecting surface: Protocol and analysis,”IEEE Trans. Wireless Commun., vol. 23, no. 11, pp. 16 527– 16 543, Nov. 2024. 13
2024
-
[39]
Deep joint source- channel coding for wireless image transmission,
E. Bourtsoulatze, D. Burth Kurka, and D. G ¨und¨uz, “Deep joint source- channel coding for wireless image transmission,”IEEE Trans. Cogn. Commun. Netw., vol. 5, no. 3, pp. 567–579, Sept. 2019
2019
-
[40]
YOLOv5: Open source neural net- work for object detection,
G. Jocher and Ultralytics, “YOLOv5: Open source neural net- work for object detection,”GitHub Repository, 2020, available: https://github.com/ultralytics/yolov5
2020
-
[41]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” inEuropean conference on computer vision (ECCV). Springer, 2014, pp. 740–755
2014
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.