Pith. sign in

REVIEW 4 major objections 5 minor 15 references

Measurement-based Evaluation of CNN-based Detection and Estimation for ISAC Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A convolutional neural network trained only on synthetic channel data can detect and estimate a real UAV's delay and Doppler in outdoor ISAC measurements, with detection probabilities of 0.48 to 0.55 and delay RMSE of about 16 to 18 ns.

desk verdict A real sim-to-real transfer evaluation with RTK ground truth that is worth citing for its data, but whose quantitative claims are weakened by conditional RMSE, missing false-alarm analysis, and inconsistent training specs. read the letter →

arxiv 2507.01799 v1 pith:EST74FLS submitted 2025-07-02 eess.SP

Steffen Schieler , Sebastian Semper , Christian Schneider , Reiner Thomä This is my paper
classification eess.SP
keywords ISACCNNtargetdetectiondelay-Dopplerestimationsynthetic-to-measurementtransferOFDMradarsensingUAVmeasurement-basedevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a CNN trained exclusively on synthetic channel data can be applied to real measurement data for joint detection and estimation of sensing targets in ISAC systems. The authors train the network on randomly generated delay–Doppler snapshots from a parametric signal model, then evaluate it on a suburban outdoor measurement of a UAV with RTK-based ground truth. They report detection probabilities of 0.48–0.55 across three receiver links and RMSEs of about 16–18 ns in delay and 8–10 Hz in Doppler, with estimates visibly aligned to ground truth. If the transfer holds, it would remove the need for costly, site-specific labeled measurement data in training such sensing models.

What carries the argument

The carrying mechanism is the CNN architecture plus its deterministic preprocessing chain, originally introduced in [5]. Preprocessing converts raw channel observations into six-channel delay-Doppler images using a pulse-pair filter, orthogonal DPSS (Discrete Prolate Spheroidal Sequences) multitaper windows, a two-dimensional DFT, and a log-magnitude/phase mapping; the CNN then maps these images to estimates of delay and Doppler for an unknown number of paths. The synthetic training set is generated by randomly sampling the number of paths, their delays, Dopplers, magnitudes, phases, and the noise level, so that the network learns a mapping from noisy delay-Doppler patterns to target parameters rather than memorizing a fixed scenario. That combination is what allows the model to be evaluated on real measurements of a different environment from a single-target UAV flight.

What would settle it

Compute the per-snapshot SNR of the UAV return in the recorded data; if the CNN detects the UAV only in snapshots whose SNR falls in the upper part of the synthetic training range, rather than across the full range, the claim of broad-SNR transfer would be undermined.

Watch

Extended reading notes

Core claim

The central claim is that a convolutional neural network trained on synthetic data from a physics-inspired channel model can detect and estimate real radar targets in an OFDM-based ISAC measurement. The paper demonstrates this by training the CNN on randomly generated snapshots with an unknown number of paths, broad SNR range, and uniform delay/Doppler parameters, and then applying it without retraining to three receiver links from a suburban outdoor UAV measurement. Using ground truth delay and Doppler computed from RTK positions, the authors compute a detection probability of 0.48–0.55 and RMSE of 16.2–18.2 ns in delay and 7.9–10.4 Hz in Doppler. The point estimates cluster on the delay-Doppler peaks of both the UAV and automotive targets of opportunity, which the paper takes as evidence that the synthetic-trained model transferred successfully to measurement data.

Load-bearing premise

The approach works only if the synthetic channel data used for training is sufficiently similar to the real measured channel, a similarity the paper states as a requirement but does not measure directly.

Editorial extensions

If this is right

  • A CNN trained on synthetic data alone can be dropped into a real ISAC receiver and produce usable target detections without site-specific retraining.
  • The reported RMSE of roughly 16–18 ns in delay is close to the 12.5 ns delay resolution of the 80 MHz measurement signal, indicating the CNN partially achieves super-resolution.
  • The method handles an unknown number of propagation paths, since the synthetic data generator samples the model order randomly, which is required for realistic clutter-rich environments.
  • The drop in detection probability when the UAV leaves the antenna main beam shows that the CNN's performance is bounded by the same front-end directivity limits as conventional radar processing.
  • Extending the same network to classify targets via micro-Doppler or spectral features is a stated next step, since the delay-Doppler estimates already isolate target returns.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's two descriptions of the synthetic training distribution (Table I versus Section II.C) differ in the number of paths and the SNR range; retraining under each setting would reveal how sensitive the transfer is to those specific choices.
  • Because the measurement SNR of the UAV returns is not reported, the results leave open which part of the synthetic SNR range actually does the work; measuring the UAV peak power relative to noise would let future work condition the detection probability on SNR.
  • The qualitative detection of automotive targets of opportunity suggests the method is not limited to the single UAV and could be scored against a multi-target ground truth if one were available.
  • A direct distribution-distance check between synthetic and measured preprocessed snapshots would convert the stated similarity requirement from an assumption into a measurable quantity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates a CNN trained on synthetic OFDM channel data for joint delay-Doppler detection and estimation, using real outdoor measurement data from the isac-uav dataset. The CNN is trained entirely on synthetic snapshots and tested on measured channels, with RTK-derived UAV positions providing independent ground truth for delay and Doppler. After filtering estimates through a groundtruth gate, the authors report detection probabilities around 0.48 to 0.55 and RMSE values of roughly 16 to 18 ns in delay and 8 to 10 Hz in Doppler, and conclude that the synthetically trained approach transfers to measurement data and is suitable for joint detection and estimation in ISAC systems.

Significance. The manuscript addresses an important practical bottleneck: machine-learning sensing methods are often only evaluated on synthetic data, so external validation on real measurements is genuinely valuable. The use of a public measurement campaign with independent RTK ground truth is a clear strength, and the qualitative alignment of estimates with measured delay-Doppler peaks in Figs. 3 to 5 is encouraging. However, the quantitative claims rest on a conditional evaluation protocol, an unquantified synthetic-to-measurement similarity assumption, and no comparison with classical baselines. If the requested revisions are made, this would be a useful benchmark for sim-to-real transfer in ISAC sensing; in its current form, the central claim is plausible but not fully established.

major comments (4)
  1. [Section II.C / Table I] The synthetic training distribution is specified inconsistently: Table I states SNR in [0, 50] dB and P ~ U[1, 10], while Section II.C states SNR in [-30, 50] dB and P ~ U[1, 30]. Because the central transfer claim rests on the synthetic data being representative of the measurement operating point, this discrepancy is load-bearing. Please correct the specification and report the actual SNR range, path-count range, and the measured SNR/operating point of the evaluation data.
  2. [Section IV.B / Eqs. (7)-(8) / Table II] The quantitative evaluation is conditional on the groundtruth filter: PD is the fraction of snapshots with at least one estimate inside the gate, and RMSE is computed only over estimates inside the gate. This makes PD not a false-alarm-aware detection probability and makes the RMSE optimistic. Please report the raw numbers, including the total number of detections outside the gate, the number of snapshots Nmeas and N∅ per Rx, and ideally a detection/false-alarm tradeoff (e.g., ROC) or at least the false-alarm rate.
  3. [Section V] No baseline comparison is provided; the conclusion explicitly defers comparison with CFAR and iterative maximum likelihood to future work. Without a baseline, the claim that the CNN is 'suitable' for joint detection and estimation in ISAC cannot be assessed. Please add a classical detection/estimation baseline (e.g., CFAR thresholding plus peak picking) and report the same PD/RMSE metrics for it.
  4. [Section II.C / Section IV] The 'sufficiently similar' premise for sim-to-real transfer is asserted but never quantified. No measured SNR is reported, no comparison of the measured delay-Doppler distribution to the synthetic prior is given, and no check of the number of significant paths versus the trained range is provided. Given that the measurement scenario contains strong line-of-sight/static clutter and targets of opportunity, the network may be evaluated out-of-distribution. Please quantify the operating point (SNR, path count, clutter structure) or provide a domain-shift/ablation analysis (e.g., retraining at the measured SNR) to support the transfer claim.
minor comments (5)
  1. [Fig. 3 caption] The word 'Cummulative' should be 'Cumulative'.
  2. [Section II.A] The notation 'θp = {γp ηp}' should be 'θp = {γp, ηp}', and 'η[i] = [ τ [i]α[i]]T' is missing a comma between the two components.
  3. [Fig. 3 caption] The phrase 'quantitative proof' overstates what the figure shows; the figure is qualitative evidence, so please rephrase.
  4. [Table I] The entry 'Trainingset Size 200 × 103' is ambiguous; please write it as 200 × 10^3 or 200k.
  5. [Section III / Table I] The measurement uses Nf = 1280 subcarriers, but Table I lists Nf = 1024 for the network input; please clarify how the 1280-subcarrier measurements are cropped, resampled, or otherwise reduced to the network input size.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CNN is trained on synthetic data and is evaluated against independent RTK groundtruth on real outdoor measurements, so the central claim is externally tested.

full rationale

The paper's central claim is that a CNN trained only on synthetic channel data can detect and estimate delay and Doppler on real outdoor measurements. The evaluation chain is external: network parameters are learned from synthetic data generated from the signal model in Eq. (2), and the reported detection probability and RMSE are computed from the CNN's outputs on measured data using an independent RTK-based groundtruth (Eqs. (4)-(5), (7)-(8), Table II). No measured result is used to set the network weights, the thresholds eps_tau and eps_alpha are derived from the sampling grid rather than fitted to the data, and the groundtruth filter is a standard gating operation. The reliance on the authors' prior architecture [5] is a self-citation, but it is not load-bearing in a circular sense: the measurement-based evaluation is new, and the architecture's output is precisely what is being tested against external groundtruth. The paper's own Section II.C notes that the synthetic data 'must be sufficiently similar' to the measurement data, but this is an unquantified domain-transfer assumption; a failure of this assumption would make the method fail empirically, not make the result true by construction. Likewise, the internal inconsistency between the SNR/path-count ranges in Table I and Section II.C is a reproducibility or correctness concern, not a circularity. The derivation chain is therefore self-contained against external benchmarks, and no step reduces by definition or by self-citation to its own inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical entities. Its central claim rests on hand-chosen synthetic data distributions, evaluation thresholds, and the assumption that synthetic and real channel statistics are similar enough for transfer. The internal inconsistencies in the reported SNR and path-count ranges further weaken this foundation. The CNN's 1.3 million parameters are learned from synthetic data during training, not fit to the measurement.

free parameters (4)
  • Groundtruth filter thresholds epsilon_tau and epsilon_alpha = epsilon_tau = 37.5 ns, epsilon_alpha = 93.75 Hz
    Chosen by hand as 3 sampling bins in Eq. (7). They define which estimates count as detections and which enter the RMSE, so they directly control both headline metrics. No sensitivity analysis is provided.
  • Synthetic data distribution ranges = tau_max = 0.02, alpha_max = 0.05, magnitudes U[0.001,1], path count U[1,10] or U[1,30] (inconsistent)
    Hand-picked to approximate the expected suburban scenario. Transfer from synthetic to measurement depends on these ranges matching the real environment, but no statistical match is demonstrated.
  • Training SNR range = 0 to 50 dB (Table I) or -30 to 50 dB (Section II.C)
    The paper states conflicting ranges. The noise level distribution during training affects how the network behaves at the real measurement's SNR, which is not reported.
  • DPSS window parameters = NW = 2, Nw = 3
    Chosen by hand for preprocessing; influences the input features and hence the learned representations that must transfer.
assumptions (6)
  • domain assumption Narrowband assumption B << fc for the signal model in Eq. (2)
    The received signal is modeled as a sum of planar waves with per-path delay, Doppler, and complex amplitude. The measurement satisfies B=80 MHz << fc=3.75 GHz, but the model ignores diffuse scattering and hardware nonlinearities.
  • domain assumption Specular multipath channel with an unknown, discrete number of paths
    Eq. (2) represents the channel as P specular paths. Real outdoor channels include diffuse components and extended targets like the UAV, which may not be perfectly point-like.
  • domain assumption Complex, zero-mean, uncorrelated Gaussian noise in Eq. (3)
    The noisy observation Y = H + N assumes Gaussian noise. Real receiver noise and channel estimation errors may be correlated or non-Gaussian, affecting transfer.
  • domain assumption RTK ground truth positions and the analytic delay/Doppler formulas (4) and (5) are accurate
    All quantitative metrics are computed against these references. Any RTK error, clock offset, or geometric approximation propagates directly into the reported PD and RMSE.
  • domain assumption The CNN architecture and postprocessing from [5] are suitable for this measurement task
    The paper does not re-validate the network design; it adopts the architecture and the remaining processing steps from the authors' prior work [5].
  • domain assumption The synthetic data distribution is sufficiently similar to the real measurement distribution
    Section II.C states this requirement but provides no quantitative comparison between synthetic and measured channel statistics. This is the load-bearing premise for sim-to-real transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measurement-based Evaluation of CNN-based Detection and Estimation for ISAC Systems." pith.science (2026). https://pith.science/paper/EST74FLS

@misc{pith2026250701799,
  author       = {Pith},
  title        = {Pith review of: Measurement-based Evaluation of CNN-based Detection and Estimation for ISAC Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EST74FLS}},
  note         = {Machine review of arXiv:2507.01799}
}
read the original abstract

In wireless sensing applications, such as ISAC, one of the first crucial signal processing steps is the detection and estimation targets from a channel estimate. Effective algorithms in this context must be robust across a broad SNR range, capable of handling an unknown number of targets, and computationally efficient for real-time implementation. During the last decade, different Machine Learning methods have emerged as promising solutions, either as standalone models or as complementing existing techniques. However, since models are often trained and evaluated on synthetic data from existing models, applying them to measurement is challenging. All the while, training directly on measurement data is prohibitive in complex propagation scenarios as a groundtruth is not available. Therefore, in this paper, we train a CNN approach for target detection and estimation on synthetic data and evaluate it on measurement data from a suburban outdoor measurement. Using knowledge of the environment as well as available groundtruth positions, we study the detection probability and accuracy of our approach. The results demonstrate that our approach works on measurement data and is suitable for joint detection and estimation of sensing targets in ISAC systems.

Figures

Figures reproduced from arXiv: 2507.01799 by the authors.

Figure 1
Figure 1. Network architecture with preprocessing, trainable blocks, and postprocessing. Lower parts detail the steps in the upper part. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Measurement scenario. A basestation uses a Tx (red) and three [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Cummulative Delay-Doppler distributions from the three Rxs. Clearly visible are two distinct tracks from automotive targets of opportunity nearby [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Estimation results for the UAV at the three Rxs after the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Estimations for all Rx over time. The delay estimates [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages

  1. [1]

    Robust DoA Estimation Using Denoising Autoencoder and Deep Neural Networks,

    D. Chen, S. Shi, X. Gu, et al., “Robust DoA Estimation Using Denoising Autoencoder and Deep Neural Networks,” IEEE Access, 2022, Conference Name: IEEE Access. DOI: 10.1109/ ACCESS.2022.3164897

    work page arXiv 2022

  2. [2]

    Direction-of-Arrival Estimation Based on Deep Neural Networks With Robustness to Array Imperfections,

    Z.-M. Liu, C. Zhang, and P. S. Yu, “Direction-of-Arrival Estimation Based on Deep Neural Networks With Robustness to Array Imperfections,” IEEE Transactions on Antennas and Propagation, no. 12, 2018. DOI: 10.1109/TAP.2018.2874430

    work page arXiv 2018

  3. [3]

    Deep Networks for Direction-of-Arrival Estimation in Low SNR,

    G. Papageorgiou, M. Sellathurai, and Y . Eldar, “Deep Networks for Direction-of-Arrival Estimation in Low SNR,” IEEE Trans- actions on Signal Processing , 2021. DOI: 10.1109/TSP.2021. 3089927

    work page doi:10.1109/tsp.2021 2021

  4. [4]

    SubspaceNet: Deep Learning-Aided Subspace Methods for DoA Estimation,

    D. H. Shmuel, J. P. Merkofer, G. Revach, et al., “SubspaceNet: Deep Learning-Aided Subspace Methods for DoA Estimation,” IEEE Transactions on Vehicular Technology , 2024. DOI: 10. 1109/TVT.2024.3496119

    arXiv 2024

  5. [5]

    Grid-free Harmonic Retrieval and Model Order Selection using Deep Convolutional Neural Networks,

    S. Schieler, S. Semper, R. Faramarzahangari, et al., “Grid-free Harmonic Retrieval and Model Order Selection using Deep Convolutional Neural Networks,” en, in 2024 18th European Conference on Antennas and Propagation (EuCAP) , 2024

    work page 2024

  6. [6]

    Cooperative Passive Coherent Location: A Promising 5G Service to Support Road Safety,

    R. S. Thom ¨a, C. Andrich, G. D. Galdo, et al., “Cooperative Passive Coherent Location: A Promising 5G Service to Support Road Safety,” IEEE Communications Magazine , no. 9, 2019, Conference Name: IEEE Communications Magazine. DOI: 10. 1109/MCOM.001.1800242

    work page 2019

  7. [7]

    Feasibility of Non-Line-of-Sight Integrated Sensing and Communication at mmWave,

    P. Tosi, M. Henninger, L. G. de Oliveira, et al., “Feasibility of Non-Line-of-Sight Integrated Sensing and Communication at mmWave,” in2024 IEEE 25th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) ,

    work page

  8. [8]

    Joint Optimization of Geometric and Probabilistic Constellation Shaping for OFDM-ISAC Systems,

    B. Geiger, F. Liu, S. Lu, et al., “Joint Optimization of Geometric and Probabilistic Constellation Shaping for OFDM-ISAC Systems,” in 2025 IEEE 5th International Symposium on Joint Communications & Sensing (JC&S) , 2025. DOI: 10 . 1109 / JCS64661.2025.10880638

    arXiv 2025

Show all 15 references
  1. [9]

    Joint Radar and Communications: Architectures, Use Cases, Aspects of Radio Access, Signal Processing, and Hardware,

    V . Shatov, B. Nuss, S. Schieler, et al. , “Joint Radar and Communications: Architectures, Use Cases, Aspects of Radio Access, Signal Processing, and Hardware,” IEEE Access, 2024. DOI: 10.1109/ACCESS.2024.3383771

    2024

  2. [10]

    Adam: A Method for Stochastic Optimization,

    D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” 2014, Publisher: arXiv Version Number: 9. DOI: 10.48550/ARXIV .1412.6980

    2014 doi arXiv

  3. [11]

    Spectrum estimation and harmonic analysis,

    D. Thomson, “Spectrum estimation and harmonic analysis,” Proceedings of the IEEE , no. 9, 1982. DOI: 10.1109/PROC. 1982.12433

    1982

  4. [12]

    Measurement Testbed for Radar and Emitter Localization of UA V at 3.75 GHz,

    J. Beuster, C. Andrich, M. D ¨obereiner, et al., “Measurement Testbed for Radar and Emitter Localization of UA V at 3.75 GHz,” in 2023 17th European Conference on Antennas and Propagation (EuCAP) , IEEE, 2023. DOI: 10 . 23919 / EuCAP57121.2023.10133118

    2023

  5. [13]

    A dataset for the study of ISAC-based UA V sensing in U-Space,

    S. Schieler, C. Smeenk, R. Faramarzahangari, et al., “A dataset for the study of ISAC-based UA V sensing in U-Space,” English, 01/2024, 2024

    2024

  6. [14]

    Scalable Undersized Dataset RF Classification: Using convolutional multistage training,

    K. Youssef, G. Schuette, Y . Cai, et al., “Scalable Undersized Dataset RF Classification: Using convolutional multistage training,” IEEE Antennas and Propagation Magazine , no. 2, 2023, Conference Name: IEEE Antennas and Propagation Magazine. DOI: 10.1109/MAP.2022.3208813

    2023

  7. [2024]

    DOI: 10.1109/SPAWC60668.2024.10694426

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.