REVIEW 4 major objections 6 minor 20 references
Low-Complexity Recurrent Neural Network Detector for Faster-than-Nyquist Signaling
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A 25-parameter Elman RNN detects faster-than-Nyquist signals within 0.4 dB of M-BCJR.
desk verdict A tiny Elman RNN gets within 0.1–0.4 dB of M-BCJR for FTN-BPSK with just 25–65 parameters; the result is credible, but the training/evaluation SNR mismatch and missing reproducibility details need attention. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Elman (plain) recurrent unit $h_t = \tanh(W_x y_t + W_h h_{t-1} + b)$, run in both directions in a single pass over the received samples. At each symbol the forward and backward hidden states are concatenated into a $2H$-dimensional vector and passed through a fully connected layer with a sigmoid during training and a sign operation at inference; the paper treats this concatenated state as a learned sufficient statistic of the BCJR $\alpha$ and $\beta$ recursions. The mechanism works because the FTN interference is banded and short, so the same small hidden dimension can summarize all relevant past and future symbols; the parameter count is $2H^2 + 8H + 1$, giving 25, 43, and 65 parameters for $H=2,3,4$.
What would settle it
Train the same bidirectional Elman detector on a root-raised-cosine faster-than-Nyquist channel with compression factor 0.7 or 0.6; if the bit error rate gap to M-BCJR at $E_b/N_0 = 10$ dB exceeds roughly 0.4 dB, the short-memory premise that justifies the small hidden size is falsified.
Extended reading notes
Core claim
The central claim is that a classical Elman RNN, with no gating, is sufficient as a standalone FTN detector because the FTN channel's ISI has a short finite memory of only a few symbols. The detector runs the Elman recurrence $h_t = \tanh(W_x y_t + W_h h_{t-1} + b)$ forward and backward over the received sequence in a single pass, concatenates the two hidden-state sequences per symbol, and maps them through a fully connected layer to a BPSK decision. The concatenated state is presented as a learned sufficient statistic of BCJR's forward and backward metrics. On the tested settings the network, with 25 to 65 trainable parameters, stays within 0.1–0.4 dB of M-BCJR while using no explicit division or exponential operations and lowering the LUT-based hardware cost by 38–67 percent depending on hidden size.
Load-bearing premise
The design depends on the faster-than-Nyquist channel's interference having a memory short enough that two to four hidden states can capture all relevant past and future information, including the colored noise at the receiver.
Editorial extensions
If this is right
- For BPSK FTN at $\tau=0.8$ and $\tau=0.9$ over AWGN, the bidirectional Elman detector can replace the M-BCJR trellis with a BER penalty of only about 0.1–0.4 dB.
- The LUT hardware cost drops by 38–46 percent for the main configurations and by up to 67 percent for $H=2$ at $\tau=0.9$, so the receiver can trade a small performance loss for a large complexity saving.
- Because the detector needs no explicit division or exponential functions, its critical path is shorter and its numerical behavior is more stable than the M-BCJR branch metric computation.
- The single-pass bidirectional processing makes the per-symbol cost independent of block length, so the same network scales to arbitrarily long frames without windowing.
- Gating mechanisms such as LSTM and GRU are unnecessary for FTN detection at these compression factors; a plain recurrent network captures the short-memory structure.
Reading between the lines
- The same short-memory argument suggests the detector should transfer to other channels with few-tap ISI, such as time-dispersive optical links, as long as the effective interference memory remains small.
- A natural stress test is lowering the compression factor: the hidden size $H$ needed to stay within 0.1–0.4 dB of M-BCJR should grow roughly with the one-sided ISI length, mapping out a complexity-versus-spectral-efficiency tradeoff curve.
- Since the network learns the colored-noise statistic from data, it may handle non-white noise without explicit covariance estimation; showing this outside AWGN would extend the claim beyond the current simulations.
- Quantizing the weights, which the paper lists as future work, would most likely reduce the LUT cost further and make the approach competitive with fixed-kernel CNN detectors at equal precision.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a bidirectional Elman recurrent neural network detector for BPSK faster-than-Nyquist signaling. The detector processes the matched-filter output once in each direction and maps the concatenated hidden states to bit decisions, with 25–65 trainable parameters. Simulations for tau=0.8 and tau=0.9 report BER within about 0.1–0.4 dB of an M-BCJR reference, with 38–46% lower LUT-based hardware cost (up to 67% for a smaller hidden size) and no explicit division or exponential operations. The authors claim this is the first Elman-RNN-based FTN detector.
Significance. If the results are reproducible, the contribution is a useful low-complexity detector for FTN-BPSK: the architecture is simple, the parameter count is explicitly quantified, the single-pass bidirectional structure is a natural analogue of the BCJR recursions, and the absence of division/exponential arithmetic is attractive for hardware. Strengths include the explicit parameter formula (2H^2+8H+1), the LUT costing methodology, and the numerical BER table that supplements the curves. The main load-bearing concerns are the mismatch between the training SNR range and the evaluation range, the unproven "sufficient statistic" claim for a small hidden state, the unspecified M-BCJR configuration, and the lack of error bars or trial counts for the low-BER region. These issues need to be resolved before the near-optimality and cost-reduction claims can be accepted.
major comments (4)
- [Section III-C, Fig. 3, Table I] The training data are generated from the 7–10 dB SNR range (Section III-C), while Fig. 3 and Table I report BER from 0–10 dB. Since the trained network has no SNR input, the weights cannot adjust the decision boundary to the noise level, so the low-SNR half of the BER curves is an extrapolation rather than a measured near-optimal behavior. Please provide matched-SNR training results (for example, training separately at each evaluation SNR, or training over the full 0–10 dB range) or an ablation demonstrating that the 0–4 dB performance is robust to the training SNR.
- [Section III-B, Eq. (10)] The statement that the concatenated hidden state "acts as a learned sufficient statistic of the BCJR α/β metrics" is asserted without proof or citation. An Elman RNN with hidden size H=2–4 has very limited state capacity, and the colored noise at the matched-filter output may create memory beyond the nominal ISI length N. Please provide a formal argument or empirical evidence (for example, BER versus hidden size saturation, or a comparison against a detector with an explicit noise-whitening front end) to justify this load-bearing assumption.
- [Section IV, Table I and Fig. 3] The M-BCJR reference configuration is not specified: the number of survivors M, the truncation length, whether the colored-noise covariance is included in the branch metric, and the block length used for counting errors are all absent. Without these details, and without error bars or the number of channel realizations, the reported 0.1–0.4 dB gaps and the low-BER values around 1e-5 to 1e-6 cannot be distinguished from Monte Carlo noise. Please provide the reference configuration and confidence intervals.
- [Table III] The hardware-cost comparison uses LUT unit weights from the authors' prior paper [13], but the table does not break down the M-BCJR cost (for example, how many division and exponential operations are counted) or state whether the same LUT unit weights were applied to the M-BCJR operations. Since the claimed 38–67% reduction is a central result, please provide a per-operation breakdown and justify the unit weights.
minor comments (6)
- [Section III-B] The term "single pass" is potentially confusing because Eqs. (8) and (9) require two traversals, one forward and one backward; please clarify that each direction is processed once.
- [Table III] The column formatting in Table III is hard to read (for example, "yes594 750" lacks a visible separator); please format the LUT counts as numeric entries with explicit column headings.
- [Section II] The noise autocorrelation is stated to be proportional to the ISI response X; please give the exact relation (for example, E[w(nτT)w(mτT)] = N0 x((n−m)τT)/2) to make the colored-noise model precise.
- [Section IV] The test-set size or number of independent trials is not reported; please add this information to Table II or to the simulation text so that the BER values can be interpreted statistically.
- [Section III-C] The statement that the per-symbol operation count is independent of block length refers to inference; during training with truncated backpropagation through time the cost may differ, so please clarify that the complexity claim is for inference.
- [Section III-C] The phrase "using no explicit division or exponential operations" should be clarified, because tanh and sigmoid are implemented with look-up tables; the claim is that the arithmetic datapath does not contain these operations, not that no nonlinear function approximations are used.
Circularity Check
No significant circularity in the BER derivation; one minor self-citation in the LUT hardware-cost model.
-
other
[Section III-C (Computational Complexity) and Table III]
"The hardware cost is evaluated using the 10-bit look-up-table (LUT) unit weights from [13] (multiplication: 113, addition: 10, division: 236, exponential: 73, tanh and sigmoid: 1 LUT)."
Reference [13] is the authors' own prior GRU-detector paper (Tokluoglu, Cicek, Cavus, Bedeer, Yanikomeroglu), with two of the present authors as co-authors. The claimed 38-67% LUT reductions are computed from these self-cited unit weights rather than from an independently derived or externally verified cost model. The hardware-savings claim is therefore partly self-referential in its cost accounting. This does not affect the BER comparison, which is an independent empirical result against the external M-BCJR benchmark, so the circularity is minor and not load-bearing for the central detection claim.
full rationale
The central claim, near-M-BCJR BER with 25-65 trainable parameters, is an empirical comparison against the external M-BCJR algorithm. The network is trained with binary cross-entropy on transmitted-bit labels, not on M-BCJR outputs and not on the reported test points, so the BER curves are not forced by construction. The 'learned counterpart of BCJR' phrasing is an analogy, not a derivation, and no fitted parameter is renamed as a prediction. The only self-referential element is the hardware-cost comparison, which uses LUT unit weights from the authors' own prior paper [13]; the quoted percentage savings are computed from that self-cited cost model. Because the cost model is applied symmetrically to both the proposed detector and M-BCJR and does not enter the BER result, this is a minor self-citation rather than a load-bearing circular step. The training-at-7-to-10-dB and evaluating-at-0-to-10-dB issue is a possible extrapolation or correctness concern, not a circularity, since the low-SNR BER is an unforced empirical prediction.
Assumptions & free parameters
free parameters (4)
- Hidden size H =
2, 3, 4
- Training SNR range =
7-10 dB
- LUT unit weights =
mult 113, add 10, div 236, exp 73, tanh/sigmoid 1 (from Ref [13])
- Trained network weights =
25 to 65 weights, values not reported
assumptions (4)
- domain assumption The FTN intersymbol interference has a short, finite memory of a few symbols (one-sided length N), so a banded Toeplitz channel model suffices.
- ad hoc to paper A bidirectional Elman RNN with hidden dimension 2 to 4 can learn a sufficient statistic equivalent to the BCJR alpha/beta metrics.
- domain assumption The concatenated noise-plus-ISI statistics can be learned from data without explicit noise covariance estimation.
- domain assumption Standard AWGN channel and RRC pulse with roll-off 0.35 model the transmission.
Cite this review
Pith. "Pith review of Low-Complexity Recurrent Neural Network Detector for Faster-than-Nyquist Signaling." pith.science (2026). https://pith.science/paper/G3CGCO5S
@misc{pith2026260804155,
author = {Pith},
title = {Pith review of: Low-Complexity Recurrent Neural Network Detector for Faster-than-Nyquist Signaling},
year = {2026},
howpublished = {\url{https://pith.science/paper/G3CGCO5S}},
note = {Machine review of arXiv:2608.04155}
}
read the original abstract
This study proposes a low-complexity, bidirectional, single-pass Elman recurrent neural network detector for binary phase-shift keying signals transmitted with faster-than-Nyquist signaling. Since the faster-than-Nyquist intersymbol interference has a short, finite memory, the classical Elman recurrent neural network, which contains no gating mechanism, is a sufficient and parameter-efficient model. The proposed detector processes the received sequence in both forward and backward directions in a single pass, forming a learned counterpart of the optimal BCJR forward-backward recursion. Under a root-raised-cosine pulse over an additive white Gaussian noise channel, simulations for two compression factors show that the proposed detector, with only twenty-five to sixty-five trainable parameters, attains a bit error rate very close to that of the M-BCJR algorithm, while reducing the look-up-table hardware cost by thirty-eight to forty-six percent and using no explicit division or exponential operations. A more compact configuration offers up to a sixty-seven percent reduction at a small performance penalty. To the best of our knowledge, this is the first study to investigate the classical Elman recurrent neural network architecture for the faster-than-Nyquist detection problem.
Figures
Reference graph
Works this paper leans on
-
[13]
GRU- Based Sequence Detection for Faster-than-Nyquist Signaling,
O. Tokluoglu, A. Cicek, E. Cavus, E. Bedeer, and H. Yanikomeroglu, “GRU- Based Sequence Detection for Faster-than-Nyquist Signaling,”IEEE Open J. Veh. Technol., vol. 7, pp. 565–581, 2026, doi: 10.1109/OJVT.2026.3653504
-
[1]
Faster-than-Nyquist signalling,
J. E. Mazo, “Faster-than-Nyquist signalling,”Bell Syst. Tech. J., vol. 54, no. 8, pp. 1451–1462, Oct. 1975
work page 1975
-
[2]
Optimal decoding of linear codes for minimizing symbol error rate,
L. R. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for minimizing symbol error rate,”IEEE Trans. Inf. Theory, vol. 20, no. 2, pp. 284–287, Mar. 1974
work page 1974
-
[3]
A. Prlja and J. B. Anderson, “Reduced-complexity receivers for strongly narrow- band intersymbol interference introduced by faster-than-Nyquist signaling,”IEEE Trans. Commun., vol. 60, no. 9, pp. 2591–2601, Sep. 2012
work page 2012
-
[4]
Frequency-domain equalization of faster-than-Nyquist signaling,
S. Sugiura, “Frequency-domain equalization of faster-than-Nyquist signaling,” IEEE Wireless Commun. Lett., vol. 2, no. 5, pp. 555–558, Oct. 2013
work page 2013
-
[5]
E. Bedeer, M. H. Ahmed, and H. Yanikomeroglu, “A very low complexity suc- cessive symbol-by-symbol sequence estimator for faster-than-Nyquist signaling,” IEEE Access, vol. 5, pp. 7414–7422, 2017
work page 2017
-
[6]
S. Li, B. Bai, J. Zhou, P. Chen, and Z. Yu, “Reduced-complexity equalization for faster-than-Nyquist signaling: New methods based on Ungerboeck observation model,”IEEE Trans. Commun., vol. 66, no. 3, pp. 1190–1204, Mar. 2018
work page 2018
-
[7]
M.-S. Baek, W. Park, and Y .-T. Lee, “Deep learning-based signal detection technique for FTN signaling-based emergency alert communication system,” in Proc. IEEE Int. Symp. Broadband Multimedia Syst. Broadcast. (BMSB), Chengdu, China, 2021, pp. 1–3
work page 2021
Show all 20 references
-
[8]
Receiver design for faster-than-Nyquist signaling: Deep-learning-based architectures,
P. Song, J. Pan, Y . Xie, and W. Tang, “Receiver design for faster-than-Nyquist signaling: Deep-learning-based architectures,”IEEE Access, vol. 8, pp. 68866– 68873, 2020
2020
-
[9]
A deep learning-aided detection method for FTN-based NOMA,
J. Pan, X. Luo, Y . Chen, and S. Liu, “A deep learning-aided detection method for FTN-based NOMA,”Wireless Commun. Mobile Comput., 2020, pp. 1–11, Jan. 2020
2020
-
[10]
Deep learning-based list sphere decoding for faster-than- Nyquist (FTN) signaling detection,
S. Abbasi and E. Bedeer, “Deep learning-based list sphere decoding for faster-than- Nyquist (FTN) signaling detection,” inProc. IEEE Veh. Technol. Conf. (VTC2022- Spring), Helsinki, Finland, 2022, pp. 1–6
2022
-
[11]
FTN-based non-orthogonal signal detection technique with machine learning in quasi-static multipath channel,
M.-S. Baek, W. Park, and Y .-T. Lee, “FTN-based non-orthogonal signal detection technique with machine learning in quasi-static multipath channel,”IEEE Trans. Broadcast., vol. 70, no. 1, pp. 78–86, Mar. 2024
2024
-
[12]
A novel sum-product detection algorithm for faster-than-Nyquist signaling: A deep learning approach,
B. Liu, S. Li, Y . Xie, and J. Yuan, “A novel sum-product detection algorithm for faster-than-Nyquist signaling: A deep learning approach,”IEEE Trans. Commun., vol. 69, no. 9, pp. 5975–5987, Sep. 2021
2021
-
[14]
Faster-than-Nyquist equaliza- tion with convolutional neural networks,
B. De Filippo, C. Amatetti, and A. Vanelli-Coralli, “Faster-than-Nyquist equaliza- tion with convolutional neural networks,” inProc. IEEE 36th Int. Symp. Personal, Indoor and Mobile Radio Commun. (PIMRC), Istanbul, Turkiye, 2025, pp. 1–6
2025
-
[15]
A novel CNN-based standalone detector for faster-than-Nyquist signaling,
O. Tokluoglu, E. Cavus, E. Bedeer, and H. Yanikomeroglu, “A novel CNN-based standalone detector for faster-than-Nyquist signaling,”IEEE Trans. Commun., vol. 73, no. 12, pp. 14316–14331, Dec. 2025
2025
-
[16]
Finding structure in time,
J. L. Elman, “Finding structure in time,”Cognitive Science, vol. 14, no. 2, pp. 179–211, 1990
1990
-
[17]
A novel domain- aware CNN architecture for faster-than-Nyquist signaling detection,
O. Tokluoglu, E. Cavus, E. Bedeer, and H. Yanikomeroglu, “A novel domain- aware CNN architecture for faster-than-Nyquist signaling detection,” inProc. IEEE 36th Int. Symp. Personal, Indoor and Mobile Radio Commun. (PIMRC), Istanbul, Turkiye, 2025, pp. 1–6
2025
-
[18]
A novel position-aware attention-enhanced Bi-GRU detector for faster-than-Nyquist signaling,
O. Tokluoglu, E. Cavus, E. Bedeer, and H. Yanikomeroglu, “A novel position-aware attention-enhanced Bi-GRU detector for faster-than-Nyquist signaling,”IEEE Com- mun. Lett., vol. 30, pp. 2317–2321, 2026, doi: 10.1109/LCOMM.2026.3704639
2026
-
[19]
Self-attention transformer-based detector for faster-than-Nyquist signaling,
N. Safak, O. Tokluoglu, and E. Cavus, “Self-attention transformer-based detector for faster-than-Nyquist signaling,” arXiv:2607.20745, 2026
2026 arXiv
-
[20]
A comparative analysis of MLP and Kolmogorov–Arnold networks (KAN) for faster-than-Nyquist signaling detection,
S. Ertan, O. Tokluoglu, and E. Cavus, “A comparative analysis of MLP and Kolmogorov–Arnold networks (KAN) for faster-than-Nyquist signaling detection,” arXiv:2608.02062, 2026
2026 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.