Pith. sign in

REVIEW 5 minor 1 cited by

Recent Advances on Machine Learning-aided DSP for Short-reach and Long-haul Optical Communications

T0 review · 0 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For probabilistically shaped coherent QAM, a variational-autoencoder blind equalizer outperforms CMA, and for short-reach IM/DD at 40 GBd, a complexity-optimized CNN outperforms linear and Volterra equalizers under a fixed MAC budget.

desk verdict Self-review of the authors' own ML equalizer work: honest and clear, but no new results and too narrow for the title. read the letter →

arxiv 2411.10101 v1 pith:FG5OTO2O submitted 2024-11-15 eess.SP cs.LG

Laurent Schmalen , Vincent Lauinger , Jonas Ney , Norbert Wehn , Patrick Matalla , Sebastian Randel , Alexander von Bank , Eike-Manuel Edelmann This is my paper · ORCID
classification eess.SPcs.LG
keywords machinelearningequalizationopticalcommunicationsvariationalautoencoderblindconvolutionalneuralnetworkFPGAimplementationspikingintensity-modulationdirectdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that machine-learning-based equalizers are ready to compete with, and in constrained settings outperform, classical DSP in optical communications. It reports two main results: a variational-autoencoder-derived blind cost function that beats a CMA-based equalizer for probabilistically shaped 64-QAM on a coherent link, and a convolutional-neural-network equalizer whose structure is optimized under a fixed multiply-accumulate budget that beats linear and Volterra equalizers in a short-reach IM/DD link at 40 GBd. It also surveys spiking-neural-network equalizers as an energy-efficient alternative, including a digital emulation that reportedly uses less energy than a classical digital equalizer. A sympathetic reader would take the paper as evidence that ML equalizers can be practical, provided the cost function matches the channel or the hardware budget is part of the design.

What carries the argument

The machinery that carries the first claim is the variational-autoencoder blind cost function, a lower bound on the log-likelihood of received symbols that combines a KL-divergence term between the soft-demapper posterior $q$ and the source distribution $P(\mathbf x)$ with an expected squared-error term $\mathbb{E}_q[\|\mathbf y - \mathbf h_\theta * \mathbf x\|^2]$ involving an estimated channel impulse response $\mathbf h_\theta$. This joint estimator-equalizer-demapper objective lets the demapper's soft outputs steer the equalizer taps without pilots. The machinery that carries the second claim is a design-space exploration in MAC operations per symbol: for each candidate CNN architecture the BER is measured against the number of multiply-accumulate operations, and the Pareto front is compared with linear and Volterra equalizers at the FPGA's complexity limit.

What would settle it

Transmit probabilistically shaped 64-QAM over a dispersive coherent link with strong Kerr nonlinearity at the symbol rates of Figure 1, and compare the variational-autoencoder equalizer with CMA; if the VAE equalizer no longer reaches a lower symbol-error rate than CMA, the paper's headline comparison holds only for linear channels and its nonlinear-channel promise rests on an unreported vector-quantized variant.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discoveries are two. First, replacing the usual mean-squared-error or constant-modulus cost with a variational-autoencoder cost function yields a blind equalizer that adapts using only received noisy symbols and source statistics; in simulation over a linear dispersive dual-polarization channel with probabilistically shaped 64-QAM, this VAE-based linear equalizer reaches lower symbol-error rates than CMA, which the paper reports as not converging. Second, a CNN equalizer with a carefully chosen layer configuration outperforms both linear and nonlinear Volterra equalizers in a short-reach IM/DD scenario once the number of multiply-accumulate operations per symbol is constrained by a real-time FPGA implementation; the paper shows this in a design-space exploration at 40 GBd. The paper additionally reports that a trainable NN equalizer including real-time training fits on an FPGA up to 20 GBd. The paper is explicit that the VAE cost derivation assumes a linear channel with AWGN, and for nonlinear channels it points to a vector-quantized variant without presenting results in this preprint.

Load-bearing premise

The variational-autoencoder equalizer's advantage over CMA is proven only under the assumption that the optical channel is linear and the only noise is ordinary additive white Gaussian noise; for the nonlinear distortions common in short-reach links the paper points to a different variant but gives no supporting measurements.

Editorial extensions

If this is right

  • A VAE-based blind equalizer makes probabilistically shaped QAM usable in the startup phase of coherent links, where CMA-based equalizers fail to converge.
  • The same VAE objective provides a channel impulse-response estimate at no extra cost, which the paper suggests could be used for integrated communications and sensing.
  • A CNN equalizer can be real-time implementable at 40 GBd on an FPGA while outperforming Volterra equalizers, so ML equalization is not inherently too complex for short-reach systems.
  • Real-time training of an NN equalizer on FPGA is feasible up to 20 GBd, enabling adaptive ML equalization in deployed transceivers.
  • Neuromorphic spiking-neural-network equalizers, and even digital emulations of them, are a plausible route to lower energy per bit than classical DSP.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the VAE cost function is replaced by its vector-quantized variant, the same blind-equalization principle may extend to strongly nonlinear channels such as short-reach passive optical networks with amplifier saturation; this is a testable prediction the preprint leaves open.
  • Editorial inference: the operation-count-normalized comparison used for the CNN result suggests a general benchmark for DSP: report BER against MAC operations per symbol, so linear, Volterra, CNN, and spiking equalizers can be compared on equal footing.
  • Editorial inference: the success of the complexity-constrained CNN suggests the same algorithm-hardware co-design could be applied to other receiver blocks, such as carrier phase estimation or demapping, without requiring new theory.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. This paper is a short review that presents recent advances in machine-learning-aided digital signal processing for optical communications, based on the authors' prior work. It covers three areas: (i) a variational-autoencoder (VAE) derived blind cost function for equalizers, shown to outperform CMA for probabilistically shaped QAM in a linear dispersive channel; (ii) a CNN-based equalizer for short-reach IM/DD systems that can outperform linear and Volterra equalizers under a MAC-operation constraint, with FPGA demonstrations; and (iii) energy-efficient equalizer implementations using spiking neural networks on neuromorphic hardware. The paper contains no new derivations or experimental data; it summarizes results that are fully referenced to the authors' previous publications, and it explicitly discusses the applicability limits of the VAE approach.

Significance. If the presented results are correct, the paper provides a useful and accessible summary of a coherent line of research showing that ML-based equalizers are not inherently impractical for high-speed optical transceivers. Its main strengths are the explicit scoping of the key claims: the VAE result is clearly limited to a linear channel with AWGN, and the CNN complexity comparison is stated under a specific MAC-operations constraint on a given FPGA platform. The paper is honest about these limitations and points to further work (e.g., vector-quantized VAEs for nonlinear channels). As a review, it does not establish new results, but it may serve as a valuable introduction for readers and demonstrates that the authors' claims are reproducible through the cited sources.

minor comments (5)
  1. [Section 3, first paragraph] The opening sentence 'It is a common misconception that ML-based equalizers are more computationally demanding than conventional equalizers' is too absolute; the subsequent sentences show that this statement holds only for a carefully optimized CNN in a specific IM/DD scenario under a MAC constraint. I recommend rewording to something like 'It is sometimes assumed that ML-based equalizers are necessarily more computationally demanding than conventional equalizers, but this is not always the case' to avoid giving the impression that ML equalizers are generally competitive in complexity.
  2. [Section 2, Fig. 1 right panel] The terms A(phi,y) and B(theta,phi,y) appear in the block diagram but are not defined or explained in the text. Please add a brief explanation of these quantities in the caption or in the text.
  3. [Section 3, Fig. 2 caption] The caption mentions a red dashed vertical line indicating the FPGA complexity limit, but the text does not discuss what 'feasible' and 'too complex' mean in terms of the MAC budget or the FPGA utilization. A single sentence in the text clarifying this would help the reader interpret the Pareto front.
  4. [Section 4] The section on spiking neural networks is very brief and does not discuss the trade-off between spike encoding and equalization performance, nor does it give a comparison with the earlier CNN results. Adding one or two sentences summarizing the key advantage and the main challenge of SNN equalizers would make the section more balanced.
  5. [Abstract and Introduction] The abstract is almost identical to the first sentence of the introduction; consider making the abstract more informative, for example by mentioning the two key concrete results (VAE outperforming CMA for PCS-QAM and CNN outperforming Volterra under a MAC constraint).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all substantive claims are attributed, scoped empirical results from prior published works with explicit assumptions.

full rationale

This paper is a review, not a derivation chain. Its two principal claims are: (i) a VAE-based blind equalizer outperforms a CMA-based equalizer for PCS-QAM on a linear dispersive channel (Section 2, Fig. 1), and (ii) a carefully optimized CNN can outperform linear and Volterra equalizers in a short-reach IM/DD scenario under a stated MAC-operation constraint (Section 3, Fig. 2). Both are presented as results of previously published studies, [6] and [11]-[13], which contain the underlying simulations and real-time FPGA implementations; nothing is re-derived inside this paper from its own inputs. The VAE claim is explicitly scoped: the paper states that 'the mathematical derivation of the novel VAE-based cost function relies on the assumption of a linear channel and additive white Gaussian noise (AWGN),' and Fig. 1's caption specifies a 'linear dispersive optical dual-polarization channel' at 40 and 90 GBd, so the comparison against CMA is an empirical benchmark, not a consequence of how the cost function is defined. The CNN claim is an empirical design-space exploration (Fig. 2 caption: 'exploration of the performance complexity trade-off'), where the Pareto-optimal point under the FPGA constraint is an observed result; the CNN is not fitting a parameter that is then renamed a prediction. The self-citations ([6], [11]-[13], [16]-[19]) are real evidence under the review rules because the cited works report machine-reproducible simulations and FPGA/demonstrator setups with specified parameters; they are externally falsifiable outside this paper's own fitted values. The acknowledged gap for nonlinear channels is deferred to vector-quantized VAE work [9,10] by Song et al., which is not self-authored. No equation in the paper reduces to its own input by construction, and no uniqueness theorem or ansatz is imported via self-citation. Verdict: no circularity, score 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters are introduced in this review. The central claims rest on previously published results, which are taken as given. The only stated in-paper assumption is the linear AWGN channel model for the VAE equalizer.

assumptions (2)
  • domain assumption The VAE-based cost function is derived assuming a linear channel with additive white Gaussian noise (AWGN).
    The paper states this assumption in Section 2 when discussing the derivation of the VAE-based cost function. The performance results in Figure 1 are for a linear dispersive channel, so nonlinear effects are excluded.
  • domain assumption The cited prior works by the authors are correct and representative.
    The paper's claims rely on the validity of the cited references, most of which are authored by the same group.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Recent Advances on Machine Learning-aided DSP for Short-reach and Long-haul Optical Communications." pith.science (2026). https://pith.science/paper/FG5OTO2O

@misc{pith2026241110101,
  author       = {Pith},
  title        = {Pith review of: Recent Advances on Machine Learning-aided DSP for Short-reach and Long-haul Optical Communications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FG5OTO2O}},
  note         = {Machine review of arXiv:2411.10101}
}
read the original abstract

In this paper, we highlight recent advances in the use of machine learning for implementing equalizers for optical communications. We highlight both algorithmic advances as well as implementation aspects using conventional and neuromorphic hardware.

Figures

Figures reproduced from arXiv: 2411.10101 by the authors.

Figure 1
Figure 1. Left: simulation results of the VAE-LE and the CMA for the transmission of 64-QAM with probabilistic constellation shaping (PCS) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: exploration of the performance complexity trade-off of CNN, Volterra, and linear equalizers for IM/DD transmission with pulse [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Feed-forward SNN-based equalizer for short-reach optical com [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Integrated recurrent optical spectral slicer for equalization of 100-km C-band IM/DD transmission

    physics.optics 2025-07 conditional novelty 6.0 of 10

    A custom silicon-photonic recurrent spectral filter, used as a two-node optical pre-processor with a simple digital equalizer, keeps 32-GBd PAM-4 C-band IM/DD transmission below FEC thresholds up to 100 km.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep learning,

    Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015

    2015

  2. [2]

    Roadmap on optical communications,

    E. Agrell et al., “Roadmap on optical communications,” Journal of Optics , vol. 26, no. 9, p. 093001, Jul. 2024. [Online]. Available: https://dx.doi.org/10.1088/2040-8986/ad261f

    work page doi:10.1088/2040-8986/ad261f 2024

  3. [3]

    End-to-end Optimization of Constellation Shaping for Wiener Phase Noise Channels with a Differentiable Blind Phase Search

    A. Rode, B. Geiger, S. Chimmalgi, and L. Schmalen, “End-to-end optimization of constellation shaping for Wiener phase noise channels with a differentiable blind phase search,”J. Lightw. Technol., vol. 41, no. 12, pp. 3849–3859, Jun. 2023, https://arxiv.org/abs/2212.03839

    work page Pith review arXiv 2023

  4. [4]

    Neural networks-based equalizers for coherent optical transmission: Caveats and pitfalls,

    P. J. Freire et al. , “Neural networks-based equalizers for coherent optical transmission: Caveats and pitfalls,” IEEE J. Sel. Topics Quantum Electron., vol. 28, no. 4: Mach. Learn. in Photon. Commun. and Meas. Syst., pp. 1–23, 2022

    work page 2022

  5. [5]

    Nonlinear equalization for optical communications based on entropy-regularized mean square error,

    F. Diedolo, G. B¨ocherer, M. Sch¨adler, and S. Calabr´o, “Nonlinear equalization for optical communications based on entropy-regularized mean square error,” in Proc. ECOC, 2022, pp. We2C–2

    work page 2022

  6. [6]

    Blind Equalization and Channel Estimation in Coherent Optical Communications Using Variational Autoencoders

    V. Lauinger, F. Buchali, and L. Schmalen, “Blind equalization and channel estimation in coherent optical communications using variational autoencoders,” IEEE J. Sel. Areas Commun. , vol. 40, no. 9, pp. 2529–2539, 2022, https://arxiv.org/abs/2204.11776

    work page Pith review arXiv 2022

  7. [7]

    Unsupervised linear and nonlinear channel equalization and decoding using variational autoencoders,

    A. Caciularu and D. Burshtein, “Unsupervised linear and nonlinear channel equalization and decoding using variational autoencoders,” IEEE Trans. on Cogn. Commun. Netw., vol. 6, no. 3, pp. 1003–1018, 2020

    work page 2020

  8. [8]

    Improving the Bootstrap of Blind Equalizers with Variational Autoencoders

    V. Lauinger, F. Buchali, and L. Schmalen, “Improving the bootstrap of blind equalizers with variational autoencoders,” in Proc. Opt. Fiber Commun. Conf. (OFC), San Diego, CA, USA, Mar. 2023, https://arxiv.org/abs/2301.06576

    work page Pith review arXiv 2023

Show all 19 references
  1. [9]

    Blind channel equalization using vector-quantized variational autoencoders,

    J. Song et al., “Blind channel equalization using vector-quantized variational autoencoders,”preprint, 2023, https://arxiv.org/abs/2302. 11687

    2023

  2. [10]

    Blind frequency-domain equalization using vector-quantized variational autoencoders,

    ——, “Blind frequency-domain equalization using vector-quantized variational autoencoders,” in Proc. ECOC, Glasgow, UK, Oct. 2023, https://arxiv.org/abs/2312.16003

    2023 arXiv

  3. [11]

    CNN-based equalization for communications: Achieving gigabit throughput with a flexible FPGA hardware architecture,

    J. Ney et al., “CNN-based equalization for communications: Achieving gigabit throughput with a flexible FPGA hardware architecture,” submitted to International Journal of Parallel Programming, 2024, https://arxiv.org/abs/2405.02323

    2024 arXiv

  4. [12]

    Real-time FPGA demonstrator of ANN-based equalization for optical communications,

    ——, “Real-time FPGA demonstrator of ANN-based equalization for optical communications,” inProc. ICMLCN, Stockholm, Sweden, May 2024, demo session, https://arxiv.org/abs/2402.15288

    2024 arXiv

  5. [13]

    From algorithm to implementation: enabling high-throughput CNN-based equalization on FPGA for optical communications,

    ——, “From algorithm to implementation: enabling high-throughput CNN-based equalization on FPGA for optical communications,” in Proc. SAMOS XXIII, Samos, Greece, Jul. 2023

    2023

  6. [14]

    Fully-blind neural network based equalization for severe nonlinear distortions in 112 Gbit/s passive optical networks,

    V. Lauinger et al., “Fully-blind neural network based equalization for severe nonlinear distortions in 112 Gbit/s passive optical networks,” in Proc. OFC, San Diego, CA, USA, Mar. 2024, https://arxiv.org/abs/2401.09579

    2024 arXiv

  7. [15]

    Achieving high throughput with a trainable neural-network-based equalizer for communications on FPGA,

    J. Ney and N. Wehn, “Achieving high throughput with a trainable neural-network-based equalizer for communications on FPGA,” in Proc. DSD, Aug. 2024, https://arxiv.org/abs/2407.02967

    2024 arXiv

  8. [16]

    Energy-efficient spiking neural network equalization for IM/DD systems with optimized neural encoding,

    A. von Bank, E.-M. Edelmann, and L. Schmalen, “Energy-efficient spiking neural network equalization for IM/DD systems with optimized neural encoding,” in Proc. Opt. Fiber Commun. Conf. (OFC) , San Diego, CA, USA, Mar. 2024, https://arxiv.org/abs/2312. 12909

    2024

  9. [17]

    Spiking neural network nonlinear demapping on neuromorphic hardware for IM/DD optical communication,

    E. Arnold et al. , “Spiking neural network nonlinear demapping on neuromorphic hardware for IM/DD optical communication,” J. Lightw. Technol., vol. 41, no. 11, pp. 3424–3431, 2023

    2023

  10. [18]

    Spiking neural network decision feedback equalization,

    E.-M. Bansbach, A. von Bank, and L. Schmalen, “Spiking neural network decision feedback equalization,” in Proc. WSA-SCC , Braunschweig, Germany, Feb. 2023, https://arxiv.org/abs/2211.04756

    2023 arXiv

  11. [19]

    Efficient FPGA implementation of an optimized SNN-based DFE for optical communications,

    M. Moursi, J. Ney, B. Hammoud, and N. Wehn, “Efficient FPGA implementation of an optimized SNN-based DFE for optical communications,” in Proc. IEEE MECOM, Nov. 2024, https://arxiv.org/abs/2409.08698

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.