Cutoff Rate Bounds and Constellation Shaping under Mixed Gaussian-Impulsive Noise

Mixed noise, composed of white Gaussian noise (WGN) and impulsive noise (IN), appears in numerous communication scenarios and can severely degrade system performance. In this paper, we optimize the transmitted constellation under mixed noise based on a theoretical analysis of the cutoff rate (CR). First, starting from the passband model of the mixed noise, we derive its corresponding baseband representation. Then, the baseband model is employed to obtain closed-form lower and upper bounds of the CR. A piecewise linear approximation is applied to derive efficient bounds by exploiting the algebraic properties of the integral terms. These bounds are then used as criteria to optimize the transmitted constellation points in both geometric positions and probabilistic distributions. The projected gradient method is employed to solve the optimization problem, and the convergence and properties of the solutions are analyzed. Numerical results demonstrate that the proposed CR bounds are tight and exhibit the expected asymptotic behavior. Furthermore, the optimized constellation scheme achieves a significant rate improvement compared to baselines.