{"id":"123e3eb5-8a70-4b6d-b7e5-33b845a24828","arxiv_id":"2501.07459","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SynthSoM is a 743.79 GB open synthetic dataset with aligned RF channel, mmWave radar, RGB, depth, and LiDAR data across five air-ground cooperative scenarios and multiple weather, time, density, frequency, and antenna conditions.","lead":"This paper introduces SynthSoM, a large open-source synthetic dataset that pairs wireless channel data with radar, camera, depth, and LiDAR sensing data in the same simulated air-ground traffic scenes. It is meant to give 6G and autonomous-systems researchers a common testbed for algorithms that combine sensing and communication without running expensive field campaigns.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TSTR validation is too narrow to support the full cross-modality/cross-condition claim, and the weather path-loss statement is physically inverted; cross-conditional validity is not yet established.","rationale":"The paper has real strengths: a large open-source dataset, integration of AirSim, WaveFarer, and Wireless InSite, an honest limitation statement about LiDAR weather effects, and a TSTR experiment providing some evidence for visual-to-path-loss transfer in a matched campus setting. The reader's conditional verdict is appropriate. My stress-test identifies the same underlying weakness: the quantitative validation is not commensurate with the breadth of the central claim. I partially agree with the reader's weakest_assumption because the reader emphasizes unshown transferability across modalities/scenarios and alignment, whereas I additionally flag a concrete internal inconsistency in the weather path-loss description, which is a specific and checkable instance of how the unvalidated condition axes may be wrong. If the path-loss rain/snow effect is genuinely reversed in the data, that would be a substantive correctness problem, not merely a missing experiment. I do not propose rejection because the issue may be a typographical error and because a narrower claim about the dataset could still hold. The fix is to correct or confirm the weather physics, add error bars and repeated runs to the TSTR/TRTR comparison, and either validate at least one more modality/scenario or explicitly scope the transferability claim to the validated subset.","tokens_in":16655,"tokens_out":4974,"duration_ms":55579,"concrete_test":"Reproduce the rainy/snowy path-loss comparison from the open repository. In the urban crossroad scenario at mmWave (or the exact settings used for Figs. 6(g)-(i)), fix one Tx-Rx link and a single snapshot, and read the Wireless InSite path loss under sunny, rainy, and snowy conditions with identical geometry, materials, and rain/snow models. Compute ΔPL_rain = PL_rainy - PL_sunny and ΔPL_snow = PL_snowy - PL_sunny across all snapshots. If either is systematically negative, the weather-condition data contradict the cited rain/snow attenuation physics and the central transferability claim fails for that axis. If both are positive and consistent with the cited attenuation measurements, the concern reduces to correcting the caption/text, and the remaining issue is that TSTR has not validated the radar/LiDAR/channel-matrix modalities.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SynthSoM provides a realistic, precisely aligned benchmark across five scenarios and all condition axes. The only quantitative transfer evidence is the TSTR/TRTR path-loss experiment, which uses one custom-matched campus scene, one task (RGB/depth to path loss), no error bars or repeated seeds, and none of the other modalities (mmWave radar waveforms/point clouds, LiDAR, channel matrices) or condition axes (weather, density, frequency, antenna type). That would already leave the cross-conditional claim under-supported. The gap is material, not merely formal: in Technical Validation, Figures 6(g)-(i) and the text state that 'path loss is much smaller on rainy and snowy days attributed to the obvious rain and snow attenuation at the mmWave frequency band.' Rain/snow attenuation is a loss mechanism: it increases path loss, so a simulator that yields smaller path loss in rain/snow is physically wrong. If the statement is only a caption error while the underlying data show increased path loss, then the validation section's wording must be corrected and the weather-axis credibility is still only qualitative. If the underlying data actually show decreased path loss, the weather-condition subset of SynthSoM is systematically invalid and cannot support the transferability claim. Either way, the paper's evidence does not currently establish the full claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SynthSoM, a large synthetic multi-modal sensing-communication dataset intended to support research on Synesthesia of Machines (SoM). The dataset is generated by a custom platform that integrates AirSim, WaveFarer, and Wireless InSite, with scenarios, dynamic trajectories, weather, time-of-day, agent density, frequency band, and antenna type aligned across the three simulators. It contains RF channel data (path loss and channel matrices), mmWave radar waveforms and point clouds, and non-RF sensory data (RGB images, depth maps, LiDAR point clouds), organized into five air-ground scenarios. The paper validates the dataset in two ways: a statistics-based qualitative comparison with real-world measurements and an ML-based transferability experiment (train on synthetic, test on real; train on real, test on real) for path-loss prediction from RGB and depth data. The central claim is that SynthSoM is realistic and precisely aligned enough to serve as a valid open benchmark for SoM-related algorithms.","tokens_in":16940,"tokens_out":4885,"duration_ms":47584,"significance":"If the validation were commensurate with the claims, SynthSoM would be a valuable community resource: it is openly released, large in scale, and broader in modality coverage than existing datasets such as ViWi, DeepSense 6G, and M3SC. The construction pipeline is described in enough detail to be reproducible, and the accompanying code and measurement campaign are concrete strengths. The paper also makes a useful comparison with prior datasets. However, the load-bearing validation evidence is much narrower than the dataset-level claims: only one task (path-loss prediction from RGB/depth), one matched campus scenario, and none of the other modalities or condition axes are quantitatively validated. In addition, the weather-related path-loss statement in Technical Validation is physically inconsistent as written. These gaps do not invalidate the dataset itself, but they do require either additional validation or a substantially more cautious statement of what has been demonstrated.","major_comments":[{"comment":"The TSTR/TRTR experiment validates only path-loss prediction from RGB images and depth maps in a single campus scenario whose simulation parameters are matched to the measurement setup. The dataset claim in the abstract and Usage Notes is that SynthSoM provides consistent data for cross-comparing SoM algorithms across five scenarios, multiple conditions, and several modalities (channel matrices, radar waveforms/point clouds, LiDAR). The current evidence does not support that generalization. The authors should either add quantitative validation for at least one additional modality and condition axis, or explicitly restrict the transferability claim to the validated path-loss task and scenario.","section":"Technical Validation, ML-Based Evaluation Metrics"},{"comment":"The text states: 'In comparison with sunny days, path loss is much smaller on rainy and snowy days attributed to the obvious rain and snow attenuation at the mmWave frequency band.' This is physically inverted: rain and snow attenuation are additional loss mechanisms and should increase path loss, not decrease it. If the underlying data actually show smaller path loss in rain/snow, the weather-condition subset is systematically invalid; if the data show larger path loss, the sentence and the caption must be corrected. The same paragraph also refers to 'sunny, rainy, and sunny days' where the last should be 'snowy days.' Because the weather axis is one of the paper's advertised condition dimensions, this must be resolved before the dataset-level validity claim can be accepted.","section":"Technical Validation, Figure 6(g)-(i) and accompanying text"},{"comment":"The accuracy metric is defined as A = |PLpr - PLgt| / PLgt, which is the absolute relative error, not an accuracy. The reported values of 89.28% and 90.35% are then presented as prediction accuracies. If the formula is a typo and the intended metric is accuracy = 1 - |PLpr - PLgt| / PLgt, the formula and surrounding explanation must be corrected. As written, the metric contradicts the reported numbers and makes the central transferability result impossible to interpret.","section":"Technical Validation, ML-Based Evaluation Metrics, accuracy formula"},{"comment":"The 89.28% versus 90.35% TSTR/TRTR comparison is reported without error bars, repeated runs, or statistical significance testing. Since MLP training involves stochastic initialization and minibatch order, a single run provides no measure of the variability of the 1.07 percentage point gap. The authors should report results over multiple seeds or otherwise quantify the uncertainty, or the claim that the gap is 'acceptable' is not supported.","section":"Technical Validation, ML-Based Evaluation Metrics, Figures 11(a)-(b)"},{"comment":"The 'precise alignment' of AirSim, WaveFarer, and Wireless InSite is a central design claim, but the paper provides no quantitative alignment error or consistency check. Stating that STL import preserves scale and origin, and that trajectories are imported from SUMO, is not the same as demonstrating that the multi-modal data streams are aligned at each snapshot to a specified tolerance. The authors should report at least one quantitative check, such as re-projection of 3D points into the RGB/depth images or comparison of a known reference path across simulators, or they should soften the 'precise alignment' claim accordingly.","section":"Methods, High-Fidelity Scenario Construction and Dynamic Scenario Generation"}],"minor_comments":[{"comment":"The loss function is listed as 'MESLoss'; this should be 'MSELoss' (mean squared error loss).","section":"Table 3"},{"comment":"The phrase 'a certain intelligent agent destiny' should be 'a certain intelligent agent density.'","section":"Background & Summary"},{"comment":"The sentence 'Figures 6(g)-(i) demonstrate path loss heatmaps in the urban crossroad scenario on sunny, rainy, and sunny days' should say 'sunny, rainy, and snowy days' to match the figure caption.","section":"Technical Validation, Figure 6 text"},{"comment":"Path loss values are reported in dBm, but path loss is a ratio and should be expressed in dB. The figure labels correctly use 'dB'; the text should be consistent.","section":"Technical Validation, Figure 6(a)-(b) text"},{"comment":"The references to 'Table 6' and 'Tables 4, 5' should be more explicit, e.g., 'Table 6 in [47] and Tables 4-5 in [48]', since the current phrasing is ambiguous.","section":"Technical Validation, ML-Based Evaluation Metrics"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: SynthSoM is a genuinely useful resource for the sensing-communication integration community. It is the first open dataset I know of that puts RF channel fading, mmWave radar, RGB, depth, and LiDAR into one aligned air-ground multi-link package, with weather, time-of-day, density, frequency, and antenna variations. The authors actually built the thing, released 743 GB on Figshare, and published the code. That alone earns credit.\n\nThe paper does a few things properly. It describes the integration of AirSim, WaveFarer, and Wireless InSite in enough detail to reproduce, and it is honest about the LiDAR-in-weather limitation, using LISA as an augmentation. Running a real measurement campaign at PKU and doing a TSTR/TRTR comparison is more than most synthetic dataset papers attempt.\n\nThe soft spots are not fatal but they are real. First, the TSTR experiment covers only path loss prediction from RGB+depth at one campus scenario. It does not touch radar, LiDAR, channel matrices, or any of the other four scenarios or condition axes. A single matched-scenario test cannot carry the paper's generalization that 'performance on the SynthSoM dataset can be transferred to the real world' across the whole dataset. Second, the text in Figures 6(g)-(i) states that path loss is 'much smaller' on rainy and snowy days due to rain/snow attenuation. Rain and snow add loss; path loss should increase. If the heatmaps actually show smaller loss in wet weather, the weather simulation is systematically wrong; if it's just a caption slip, it needs fixing and the weather axis is still only qualitatively supported. Either way, the current wording is a contradiction. Third, the 'accuracy' formula A = |PLpr-PLgt|/Pgt is relative error, not accuracy. Reporting 89.28% as 'accuracy' when it is actually around 10% relative error is confusing, and there are no error bars or repeated seeds. Fourth, 'precise alignment' is asserted rather than measured; a quantitative registration error would strengthen the claim.\n\nWho should read this: anyone working on multimodal sensing and communication datasets or SoM-related algorithms. It deserves a serious referee. The dataset is a contribution, but the paper needs a corrected weather statement, a clearer accuracy metric, and either more validation or a more carefully scoped claim.","headline":"Ambitious open SoM dataset with real engineering value; validation is too narrow and one weather claim is physically reversed.","tokens_in":17481,"tokens_out":2989,"would_cite":true,"duration_ms":27971,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SynthSoM is an open synthetic dataset that pairs ray-traced wireless channel data with mmWave radar, RGB, depth, and LiDAR in frame-aligned air-ground scenarios, and validates the data via sim-to-real path-loss prediction at 89.28%…","keywords":["SynthSoM dataset","multi-modal sensing-communication","Synesthesia of Machines","mmWave radar","ray-tracing simulation","path loss","LiDAR point clouds","air-ground cooperative scenario"],"falsifier":"Run the same train-on-synthetic/test-on-real protocol on a different modality from the dataset, such as mmWave radar object detection or channel-matrix prediction, across the other four scenarios, and measure the accuracy gap relative to train-on-real. If the gap exceeds about ten percentage points, or if frame-level cross-modal matching between RGB, LiDAR, and radar point clouds fails on a noticeable fraction of snapshots, the paper's transferability and precise-alignment claims would be refuted.","tokens_in":16488,"feed_emoji":"📡","tokens_out":5698,"duration_ms":49004,"temperature":0.7,"pith_summary":"This paper introduces SynthSoM, an open synthetic dataset in which wireless communication data and multi-modal sensory data are generated in the same simulated world and aligned frame by frame. The dataset spans five air-ground cooperative scenarios and combinations of weather, time of day, agent density, frequency band, and antenna type, and contains channel matrices and path loss, mmWave radar waveforms and point clouds, RGB images, depth maps, and LiDAR point clouds. The authors' central claim is that this synthetic data is realistic enough to support Synesthesia of Machines research: they show qualitative consistency with measured phenomena and report a train-on-synthetic/test-on-real path-loss prediction accuracy of 89.28% against 90.35% for train/test on real data. If the transferability claim holds, SynthSoM offers a common, customizable benchmark for studying how communications and sensing can reinforce each other.","feed_headline":"Aligned synthetic dataset couples wireless channels to sensors","feed_subtitle":"Train-on-synthetic path-loss prediction hits 89.28%, within 1.1 points of train-on-real.","key_machinery":"The load-bearing mechanism is the four-step simulation platform: high-fidelity scenario construction by importing identical STL models into AirSim, WaveFarer, and Wireless InSite; comprehensive condition simulation covering weather, time of day, agent density, frequency band, and antenna type; dynamic scenario generation in which SUMO-generated trajectories are imported into all three software; and automatic data collection and export. The alignment guarantee rests on exporting the same STL models with no rotation, scaling, or origin offset, and setting identical coordinates, a 1:1 scale, and the same world origin in each software. This is what makes the communication data, RF sensory data, and non-RF sensory data correspond snapshot by snapshot.","core_discovery":"The central discovery is that a single simulation platform can produce precisely aligned communication and multi-modal sensing data at scale, and that the resulting synthetic data transfers to the real world for a representative learning task. The platform combines AirSim for RGB, depth, and LiDAR, WaveFarer for mmWave radar, and Wireless InSite for ray-traced channel data, using identical STL models, coordinates, weather conditions, and trajectories across the three software. The validation compares train-on-synthetic/test-on-real and train-on-real/test-on-real path-loss prediction, finding accuracies of 89.28% and 90.35%, a gap under 10% that the authors treat as acceptable for a synthetic dataset. The paper therefore frames SynthSoM as a consistent open benchmark for cross-comparing, calibrating, and baselining SoM-related algorithms.","pith_inferences":["The transferability evidence is based on one task, path-loss prediction from RGB and depth in a single campus scene; extending the same train-on-synthetic/test-on-real protocol to mmWave radar point clouds or channel matrices across the other four scenarios is a direct test the authors did not report.","If alignment is as precise as claimed, a cross-modal retrieval benchmark that matches an RGB frame to its corresponding LiDAR scan and radar point cloud should achieve near-perfect accuracy, offering an inexpensive independent check of the alignment claim.","The dataset's scale of 743.79 GB may make full downloads impractical for many groups; providing scenario-specific subsets or a smaller core sample would lower the barrier to adoption and broaden the benchmark's use."],"forward_implications":["SynthSoM supplies frame-aligned channel matrices, mmWave radar, RGB, depth, and LiDAR for the same agents, enabling multi-modal models that jointly exploit communications and sensing.","The reported train-on-synthetic/test-on-real accuracy of 89.28% suggests that models trained on SynthSoM can be applied to real measurements with a small performance drop, at least for path-loss prediction.","The platform's four-step pipeline can be reused to generate new scenarios beyond the five included areas, because scenario files, conditions, and trajectories are modular inputs.","Consistent data across multiple conditions supports cross-comparison of SoM algorithms, model calibration, and transfer-learning baselines."],"supporting_citations":[{"why":"AirSim supplies the RGB, depth, and LiDAR generation with UE4 rendering and sensor noise models.","marker":"22"},{"why":"WaveFarer performs ray-traced FMCW mmWave radar simulation and range-Doppler processing for radar waveforms and point clouds.","marker":"26"},{"why":"Wireless InSite provides ray-tracing communication channel simulation for path loss and channel matrices.","marker":"27"},{"why":"KITTI is the real-world multi-modal measurement dataset that SynthSoM extends by adding communications and radar.","marker":"17"},{"why":"DeepSense 6G is the real-world multi-modal dataset that motivates the need for depth maps and weather coverage.","marker":"18"},{"why":"ViWi is the prior synthetic vision-wireless dataset that lacks UAV scenarios, radar, weather, and time-of-day conditions.","marker":"20"},{"why":"M3SC is the authors' earlier synthetic dataset that lacked large-scale fading data, real-world validation, and open-source code.","marker":"21"},{"why":"LISA provides the physics-based rain and snow augmentation used for LiDAR point clouds.","marker":"30"},{"why":"SUMO generates the realistic vehicle and UAV trajectories that are aligned across the three simulators.","marker":"34"},{"why":"The TSTR/TRTR evaluation protocol is used to measure whether synthetic training transfers to real measurements.","marker":"45"}],"fun_headline_variants":["SynthSoM: aligned synthetic data for sensing-communication","One platform yields aligned wireless and sensor data","Synthetic-to-real path loss gap under 1.1 points","Open dataset fuses RF, radar, and vision for SoM","Sim integration creates precise multi-modal comms data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole-dataset claim rests on the assumption that transferability demonstrated for path-loss prediction from RGB and depth in one campus scenario extends to every modality and every scenario-condition combination in the dataset, and that the three simulators are aligned closely enough that no quantitative registration error undermines the multi-modal correspondences.","fun_headline_variants_meta":{"raw":{"variants":["SynthSoM: aligned synthetic data for sensing-communication","One platform yields aligned wireless and sensor data","Synthetic-to-real path loss gap under 1.1 points","Open dataset fuses RF, radar, and vision for SoM","Sim integration creates precise multi-modal comms data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001201,"raw_usage":{"total_tokens":4950,"prompt_tokens":946,"completion_tokens":4004,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":3923}},"tokens_in":562,"tokens_out":4004,"duration_ms":29283,"temperature":1.0,"reasoning_tokens":3923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:40:51.871900+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same train-on-synthetic/test-on-real protocol on a different modality from the dataset, such as mmWave radar object detection or channel-matrix prediction, across the other four scenarios, and measure the accuracy gap relative to train-on-real. If the gap exceeds about ten percentage points, or if frame-level cross-modal matching between RGB, LiDAR, and radar point clouds fails on a noticeable fraction of snapshots, the paper's transferability and precise-alignment claims would be refuted.","supporting_citations":[{"cited_title":"& Kapoor, A","cited_arxiv_id":null,"evidence_quote":"AirSim supplies the RGB, depth, and LiDAR generation with UE4 rendering and sensor noise models."},{"cited_title":"Wavefarer","cited_arxiv_id":null,"evidence_quote":"WaveFarer performs ray-traced FMCW mmWave radar simulation and range-Doppler processing for radar waveforms and point clouds."},{"cited_title":"Wireless insite","cited_arxiv_id":null,"evidence_quote":"Wireless InSite provides ray-tracing communication channel simulation for path loss and channel matrices."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"M3SC is the authors' earlier synthetic dataset that lacked large-scale fading data, real-world validation, and open-source code."}],"review_version":1}