Gradient-free training of autoencoders for non-differentiable communication channels

Training of autoencoders using the back-propagation algorithm is challenging for non-differential channel models or in an experimental environment where gradients cannot be computed. In this paper, we study a gradient-free training method based on the cubature Kalman filter. To numerically validate the method, the autoencoder is employed to perform geometric constellation shaping on differentiable communication channels, showing the same performance as the back-propagation algorithm. Further investigation is done on a non-differentiable communication channel that includes: laser phase noise, additive white Gaussian noise and blind phase search-based phase noise compensation. Our results indicate that the autoencoder can be successfully optimized using the proposed training method to achieve better robustness to residual phase noise with respect to standard constellation schemes such as Quadrature Amplitude Modulation and Iterative Polar Modulation for the considered conditions.