REVIEW 4 major objections 6 minor 21 references
A Dataset Similarity Evaluation Framework for Wireless Communications and Sensing
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Dataset similarity in UMAP space predicts wireless model performance with 0.85 correlation.
desk verdict Real empirical finding, but the headline 0.85 correlation is in-sample selection and needs held-out validation before it can be trusted as a predictive claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is UMAP (Uniform Manifold Approximation and Projection), a nonlinear dimensionality-reduction method that embeds each high-dimensional dataset in a low-dimensional space while preserving local neighborhood and global structure. After embedding, the framework computes either pairwise, cluster-centroid, or average Euclidean distances, or a one-dimensional Wasserstein (earth mover) distance averaged over latent dimensions, between the embedded point clouds. It then correlates the resulting dataset-distance matrix with the performance-drop matrix of a model trained on one dataset and tested on the others, using the Pearson correlation coefficient as the score for each candidate distance metric. The UMAP embedding is the load-bearing step: it removes noise and redundancy so that simple distances in the latent space reflect the distributional differences that drive model performance.
What would settle it
Compute the proposed UMAP-based Euclidean and Wasserstein distances and the corresponding performance-drop matrices on datasets from several distinct sites, frequency bands, and channel models (for example, stochastic channels or different ray-traced scenes); if the Pearson correlations fall well below 0.85, or fluctuate widely across environments, the framework's claim of general dataset-similarity evaluation would be refuted.
Extended reading notes
Core claim
The paper's central claim is that dataset similarity, measured in a UMAP-projected latent space, tracks model generalization for an unsupervised wireless task. For the CSI compression autoencoder, distance matrices computed with Euclidean and Wasserstein distances on UMAP embeddings of 20 area-datasets show Pearson correlations above 0.85 with the matrix of performance drops (NMSE) when models trained on one area are tested on the others. This approaches the 0.94 correlation achieved by an autoencoder latent space that is task-specific but expensive to obtain, and it far exceeds the best raw-input-space correlation of about 0.55. The authors interpret this as evidence that UMAP preserves the local and global structure that determines how well a compression model transfers, making UMAP-based distances a practical, model-agnostic stand-in for actual retraining evaluation.
Load-bearing premise
The 20 datasets all come from one simulated campus scene at a single frequency, and the winning metric choices were made after seeing the correlation results on those same datasets.
Editorial extensions
If this is right
- Dataset distances can predict model performance on unseen areas without retraining, enabling fast dataset ranking.
- Practitioners can decide when retraining is needed by monitoring the distance between a deployed dataset and newly observed data.
- Synthetic datasets can be matched to real deployment data by minimizing UMAP-based distance, improving data augmentation.
- The framework provides a benchmark tool for comparing dataset realism and quality across wireless datasets.
- The approach can reduce the computational cost of dataset evaluation compared with training an autoencoder for every candidate dataset.
Reading between the lines
- The reported correlations come from 20 areas of a single simulated campus scene, so the strong result may not hold across sites, frequency bands, or other channel models; testing on multiple environments would show whether the approach generalizes.
- Because UMAP hyperparameters and the choice of distance metric were selected after inspecting correlations on the same datasets, a held-out validation split (new areas or new scenes) is needed to rule out overfitting of the metric choice.
- The same correlation machinery could be applied to other wireless tasks such as beam prediction, channel prediction, or localization, where dataset distance might predict transfer performance just as it does for CSI compression.
- A natural extension is to use the learned distance to actively select training datasets for foundation models, minimizing the need for task-specific retraining.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a framework for computing task-specific, model-agnostic dataset similarity distances and correlating them with model performance drops, targeting wireless communications and sensing. The framework is demonstrated on a channel state information (CSI) compression task using autoencoders and 20 spatial area datasets from a single ray-traced ASU campus scene. The authors design distance metrics based on UMAP dimensionality reduction followed by Euclidean and Wasserstein distances on latent-space KNN clusters, and report in Table II that these metrics achieve correlations above 0.85 with model performance, outperforming traditional distances computed in the raw input space. The paper positions this as enabling data selection, augmentation, and retraining decisions across deployment settings.
Significance. If the reported correlations are predictive rather than in-sample artifacts, the framework would be a valuable tool for dataset management in wireless ML, with practical relevance to data augmentation, benchmarking, and transfer learning. The paper is open-source and emphasizes reproducibility, which is a concrete strength. However, the quantitative claim rests on a single experimental setup (one scene, one frequency, one task), and the central correlations in Table II are obtained after selecting the dimensionality-reduction method, its hyperparameters, and the distance metric on the same 20 datasets used to evaluate the correlation. The AE latent-space column is correctly identified by the authors as an expected upper bound, but the same type of selection-bias concern is not applied to the UMAP-based results. With proper out-of-sample validation, significance testing, and error bars, the framework's contribution would be much stronger.
major comments (4)
- [Section IV-E, Table II] The UMAP configuration (method, n_neighbors, min_dist, latent dimension) and the distance metric (Euclidean vs. Wasserstein) are selected after evaluating correlations on the same 20 area-datasets used to report the headline values. This is in-sample selection: the expected maximum correlation over the tested grid under a null hypothesis of no true relationship is nonzero, so the reported 0.85 values are not estimates of predictive performance on new datasets. The paper should provide an out-of-sample validation, such as splitting the 20 datasets into selection and validation sets, cross-validating the configuration choice, or at least reporting the distribution of correlations over all configurations along with a permutation test.
- [Section IV-C, IV-F] The 20 datasets are all spatial areas from one ray-traced ASU campus scene at one frequency. They are not independent deployment environments: nearby areas share propagation geometry, materials, and user distributions, so the effective sample size is much smaller than 20. The abstract and Section I claim generality to 'different sites or frequency bands', but no such evidence is provided. The generality claim should be tempered to the tested scenario or supported with additional datasets from multiple sites/frequency bands/channel models.
- [Table II] The correlations are reported as single point estimates with no confidence intervals, standard errors, or significance levels. UMAP is stochastic, and no multiple runs are reported. Please report the mean and standard deviation over at least several UMAP runs for the final selected configuration, and provide bootstrap or permutation-based confidence intervals for the correlations, particularly for the 0.85 entries. Without this, it is impossible to distinguish the reported correlation from noise or from selection over a large configuration grid.
- [Section IV-F] The paper correctly states that the AE latent-space correlation (0.94) is unsurprising because the AE dimensionality-reduction architecture is similar to the performance model, yet it does not apply the same skepticism to the UMAP column. UMAP hyperparameters are chosen on the same datasets that define the performance correlation, so the UMAP-based metrics may also benefit from selection bias, albeit less directly than the AE. The manuscript should explicitly address this and provide evidence that the designed metric predicts performance on a held-out dataset without re-tuning the UMAP hyperparameters or distance choice.
minor comments (6)
- [Section III-B] The number of clusters K used in the cluster-based Euclidean distance is not specified; please state the value and how it is determined.
- [Section III-A] The UMAP hyperparameters (n_neighbors, min_dist, metric) are not reported; provide the exact configuration to enable reproducibility.
- [Section IV-B] The sentence 'These 20 models are used to access performance drops' should read 'assess' instead of 'access'.
- [Fig. 1 and Fig. 2 captions] The text 'speci/f_ic' appears in the figure captions; this appears to be a rendering artifact for 'specific' and should be corrected.
- [Section IV-D] The compute times in Table I are given without specifying the hardware; state the platform (CPU/GPU, single node, etc.) to make the times interpretable.
- [References] Reference [12] (Fraunhofer ISE annual report) appears unrelated to channel simulation; please replace it with an appropriate citation for a channel simulator or remove it.
Circularity Check
Headline UMAP correlation is selected in-sample on the same 20 area-datasets, and the AE latent-space column is self-referential (though disclosed as an upper bound).
-
fitted input called prediction
[Section IV-E (UMAP for Improved Distance Computation) and Table II]
"UMAP was found to be the most effective among dimensionality reduction techniques... UMAP provides a close approximation to the AE’s performance, with correlations of around 0.85 for both Euclidean and Wasserstein distances."
The UMAP method and the Euclidean/Wasserstein metrics are chosen by comparing the very correlations reported in Table II on the same 20 area-datasets. Section IV-E first selects UMAP over PCA and t-SNE, and Table II then reports the winning combinations; no held-out split, cross-validation, confidence interval, or permutation null is described. The headline 'correlations around 0.85' is therefore the selected maximum of an in-sample configuration search, not an independent estimate of how the designed metric predicts performance on new datasets. The 'prediction' step is statistically forced by the selection step because the same performance matrix is used both to choose the metric and to evaluate it.
-
self definitional
[Section IV-F (Results: Distances in Latent Space), AE as an upper bound]
"This is unsurprising, as the AE architecture used for dimensionality reduction is similar to the model used for performance evaluation."
The performance-drop matrix is defined by NMSE of autoencoders trained and tested across the 20 areas, while the AE-latent-space distances come from an autoencoder trained for the same CSI-compression task on all areas. Both sides of the correlation are thus produced by the same architecture family and the same reconstruction objective, so the 0.94 correlation measures self-consistency of the AE representation rather than an independent dataset-similarity signal. The paper explicitly labels this an upper bound and does not present it as a designed metric, so the circularity is acknowledged; still, this self-referential column is included in Table II as comparative evidence.
full rationale
The framework is not a mathematical tautology: UMAP embeddings are computed without using the autoencoder whose NMSE defines performance, so the central UMAP-based metric has independent content. However, the headline result is weakened by an in-sample selection loop. Section IV-E chooses UMAP over PCA and t-SNE, and Table II chooses Euclidean and Wasserstein distances over many alternatives, by comparing the same Pearson correlations that are then reported as the designed metrics achieving about 0.85. No validation split, confidence interval, or permutation test is reported, so 0.85 is the best value from a configuration search on the same 20 area-datasets rather than an unbiased prediction of correlation on new datasets. The AE latent-space column is explicitly self-referential: the paper notes that the high 0.94 correlation is unsurprising because the AE used for dimensionality reduction is similar to the model used for performance evaluation. That step is disclosed and treated as an upper bound, so it is less load-bearing than the UMAP claim. No load-bearing self-citation chain appears; the citations to DeepMIMO, UMAP, and CSINet+ are background or external. Overall, the central claim partially reduces to a fitted selection of the metric, giving a score of 6 rather than a fully definitional 10.
Assumptions & free parameters
free parameters (3)
- UMAP n_neighbors
- UMAP min_dist
- Number of clusters K for cluster-based distance
assumptions (3)
- domain assumption UMAP preserves task-relevant topology of wireless CSI data
- domain assumption 20 areas of the ASU campus scene are representative of wireless deployment diversity
- standard math Pearson correlation over matrix entries treats pairs as independent
Cite this review
Pith. "Pith review of A Dataset Similarity Evaluation Framework for Wireless Communications and Sensing." pith.science (2026). https://pith.science/paper/A2VG7QRC
@misc{pith2026241205556,
author = {Pith},
title = {Pith review of: A Dataset Similarity Evaluation Framework for Wireless Communications and Sensing},
year = {2026},
howpublished = {\url{https://pith.science/paper/A2VG7QRC}},
note = {Machine review of arXiv:2412.05556}
}
read the original abstract
This paper introduces a task-specific, model-agnostic framework for evaluating dataset similarity, providing a means to assess and compare dataset realism and quality. Such a framework is crucial for augmenting real-world data, improving benchmarking, and making informed retraining decisions when adapting to new deployment settings, such as different sites or frequency bands. The proposed framework is employed to design metrics based on UMAP topology-preserving dimensionality reduction, leveraging Wasserstein and Euclidean distances on latent space KNN clusters. The designed metrics show correlations above 0.85 between dataset distances and model performances on a channel state information compression unsupervised machine learning task leveraging autoencoder architectures. The results show that the designed metrics outperform traditional methods.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Deep learning for massive mimo csi feedback,
C.-K. Wen, W.-T. Shih, and S. Jin, “Deep learning for massive mimo csi feedback,” IEEE Wireless Communications Letters , vol. 7, no. 5, pp. 748–751, 2018
work page 2018
-
[2]
Enabling large intelligent surfaces with compressive sensing and deep learning,
A. Taha, M. Alrabeiah, and A. Alkhateeb, “Enabling large intelligent surfaces with compressive sensing and deep learning,” IEEE Access , vol. 9, pp. 44 304–44 321, 2021
2021
-
[3]
Deep learning coordinated beamforming for highly-mobile millimeter wave systems,
A. Alkhateeb, S. Alex, P. Varkey, Y . Li, Q. Qu, and D. Tujkovic, “Deep learning coordinated beamforming for highly-mobile millimeter wave systems,” IEEE Access , vol. 6, pp. 37 328–37 348, 2018
work page 2018
-
[4]
Deep learning for mmwave beam and blockage prediction using sub-6 ghz channels,
M. Alrabeiah and A. Alkhateeb, “Deep learning for mmwave beam and blockage prediction using sub-6 ghz channels,” IEEE Transactions on Communications, vol. 68, no. 9, pp. 5504–5518, 2020
work page 2020
-
[5]
F. B. Mismar, B. L. Evans, and A. Alkhateeb, “Deep reinforcement learning for 5g networks: Joint beamforming, power control, and inter- ference coordination,” IEEE Transactions on Communications , vol. 68, no. 3, pp. 1581–1592, 2020
work page 2020
-
[6]
Learnable wireless digital twins: Reconstructing electromagnetic field with neural representations,
S. Jiang, Q. Qu, X. Pan, A. Agrawal, R. Newcombe, and A. Alkhateeb, “Learnable wireless digital twins: Reconstructing electromagnetic field with neural representations,” 2024. [Online]. Available: https://arxiv.org/abs/2409.02564
arXiv 2024
-
[7]
Large wireless model (lwm): A foundation model for wireless channels,
S. Alikhani, G. Charan, and A. Alkhateeb, “Large wireless model (lwm): A foundation model for wireless channels,” 2024. [Online]. Available: https://arxiv.org/abs/2411.08872
arXiv 2024
-
[8]
Millimeter wave channel modeling and cellular capacity evaluation,
M. R. Akdeniz, Y . Liu, M. K. Samimi, S. Sun, S. Rangan, T. S. Rappaport, and E. Erkip, “Millimeter wave channel modeling and cellular capacity evaluation,” IEEE Journal on Selected Areas in Com- munications, vol. 32, no. 6, pp. 1164–1179, 2014
2014
Show all 21 references
-
[9]
Platforms for advanced wireless research (pawr),
“Platforms for advanced wireless research (pawr),” https://www. advancedwireless.org/, accessed: 2024-10-06
2024
-
[10]
3GPP TR 38.901 V16.1.0: Study on channel model for frequencies from 0.5 to 100 ghz,
3rd Generation Partnership Project (3GPP), “3GPP TR 38.901 V16.1.0: Study on channel model for frequencies from 0.5 to 100 ghz,” ETSI, Technical Report V16.1.0, Dec. 2019, available on- line at: https://www.etsi.org/deliver/etsi tr/138900 138999/138901/16. 01.00 60/tr 138901v1...
2019
-
[11]
A novel millimeter-wave channel simulator and applications for 5g wireless communications,
S. Sun, T. S. Rappaport, M. Shafi, J. Tang, H. Zhang, Y . X. Azar, and K. Wang, “A novel millimeter-wave channel simulator and applications for 5g wireless communications,” Proceedings of the IEEE , vol. 105, no. 12, pp. 2410–2435, 2017
2017
-
[12]
Fraunhofer ise annual report 2023- 2024,
F. ISE, “Fraunhofer ise annual report 2023- 2024,” 2024, accessed: 2024-10-05. [Online]. Avail- able: https://www.ise.fraunhofer.de/content/dam/ise/en/documents/ annual reports/fraunhofer-ise-annual-report-2023-2024.pdf
2023
-
[13]
Wireless insite ray-tracing software,
“Wireless insite ray-tracing software,” https://www.remcom.com/ wireless-insite-em-propagation-software, accessed: 2024-10-06
2024
-
[14]
Sionnart: High-fidelity ray tracing simulator for 6g research,
“Sionnart: High-fidelity ray tracing simulator for 6g research,” https: //nvlabs.github.io/sionna-ray-tracing/, accessed: 2024-10-06
2024
-
[15]
Deepmimo: A generic dataset for millimeter wave and massive mimo applications,
A. Alkhateeb, “Deepmimo: A generic dataset for millimeter wave and massive mimo applications,” arXiv preprint arXiv:1902.06435 , 2019. [Online]. Available: https://arxiv.org/abs/1902.06435
1902 arXiv
-
[16]
Umap: Uniform manifold approx- imation and projection for dimension reduction,
J. M. L. McInnes, J. Healy, “Umap: Uniform manifold approx- imation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018
2018 arXiv
-
[17]
Visualizing data using t-sne,
L. Van der Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of Machine Learning Research , 2008
2008
-
[18]
Villani, Optimal Transport: Old and New
C. Villani, Optimal Transport: Old and New . Springer-Verlag Berlin Heidelberg, 2008
2008
-
[19]
Computational optimal transport,
G. Peyr ´e and M. Cuturi, “Computational optimal transport,” F oundations and Trends in Machine Learning , vol. 11, no. 5-6, pp. 355–607, 2019
2019
-
[20]
The earth mover’s distance as a metric for image retrieval,
Y . Rubner, C. Tomasi, and L. Guibas, “The earth mover’s distance as a metric for image retrieval,” International Journal of Computer Vision , vol. 40, pp. 99–121, 11 2000
2000
-
[21]
Convolutional neural network based multiple-rate compressive sensing for massive mimo csi feedback: Design, simulation, and analysis,
J. Guo, C.-K. Wen, S. Jin, and G. Y . Li, “Convolutional neural network based multiple-rate compressive sensing for massive mimo csi feedback: Design, simulation, and analysis,” 2019. [Online]. Available: https://arxiv.org/abs/1906.06007
2019 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.