Hybrid Beamforming Design for RSMA-enabled Near-Field Integrated Sensing and Communications

Integrated sensing and communication (ISAC) networks leverage extremely large antenna arrays and high frequencies. This inevitably extends the Rayleigh distance, making near-field (NF) spherical wave propagation dominant. This unlocks numerous spatial degrees of freedom, raising the challenge of optimizing them for communication and sensing tradeoffs. To this end, we propose a rate-splitting multiple access (RSMA)-based NF-ISAC transmit scheme utilizing hybrid analog-digital antennas. RSMA enhances interference management, while a variable number of dedicated sensing beams adds beamforming flexibility. The objective is to maximize the minimum communication rate while ensuring multi-target sensing performance by jointly optimizing receive filters, analog and digital beamformers, common rate allocation, and the sensing beam count. To address uncertainty in sensing beam allocation, a rank-zero solution reconstruction method demonstrates that dedicated sensing beams are unnecessary for NF multi-target detection. A penalty dual decomposition (PDD)-based double-loop algorithm is introduced, employing weighted minimum mean-squared error (WMMSE) and quadratic transforms to reformulate communication and sensing rates. Simulations reveal that the proposed scheme: 1) achieves performance comparable to fully digital beamforming with fewer RF chains, (2) maintains NF multi-target detection without compromising communication rates, and 3) significantly outperforms conventional multiple access schemes and far-field ISAC systems.