Practical Limits of Higher-Order QAM in Hollow-Core Fiber Systems

Hollow-core fiber (HCF) is widely expected to enable higher-order quadrature amplitude modulation (QAM) because of its near-vacuum Kerr nonlinearity and higher launch power. We develop a per-channel effective signal-to-noise ratio (SNR) budget that combines, in reciprocal form, optical-link impairments including amplified spontaneous emission, Kerr nonlinear interference (NLI), inter-modal interference (IMI), pigtail NLI, and CO2 gas absorption; a parameterized, symbol-rate-dependent transceiver back-to-back SNR ceiling determined by effective-number-of-bits at rate, analog bandwidth, and Tx/Rx nonlinearity; and the remaining transceiver and line impairments, including laser phase noise, equalization-enhanced phase noise, timing jitter, polarization-dependent loss, and amplifier gain ripple with filter narrowing, each expressed as an equivalent SNR floor. The central result, for a deployment-realistic 64GBaud system with 75GHz channel spacing over 6THz, is that once HCF removes the fiber limits, the transceiver ceiling rather than the fiber sets the achievable modulation order: a roughly 25dB ceiling at 64GBaud makes 1024-QAM and above infeasible on either fiber, confining ultra-high-order QAM to low baud rates. HCF therefore provides its main advantage in reach and achievable baud rate at a given modulation order: at an IMI coefficient of kappa=-55dB/km, 256-QAM reach increases from about 45km to about 170km and 64-QAM reach from about 415km to about 2275km when moving from single-mode fiber to HCF. In the C-band, CO2 absorption lines are weak and sparse, so channels placed away from the lines follow the gas-free baseline, while only worst-case placements lose reach at long distances. In the L-band, the stronger absorption bands are denser than the channel bandwidth, making line avoidance spectrally costly, and a channel placed on a line loses one to two QAM orders.