Spiral antiferromagnets beyond the spin-wave approximation: frustrated XY and Heisenberg models in the honeycomb lattice

We examine the stability of classical states with a generic incommensurate spiral order against quantum fluctuations. Specifically, we focus on the frustrated spin-1/2 $XY$ and Heisenberg models on the honeycomb lattice with nearest-neighbor $J_1$ and next-nearest-neighbor $J_2$ antiferromagnetic couplings. Our variational approach is based on the Jastrow wave functions, which include quantum correlations on top of classical spin waves. We perform a systematic optimization of wave vectors and Jastrow pseudo-potentials within this class of variational states and find that quantum fluctuations favor collinear states over generic coplanar spirals. The N\'eel state with ${\bf Q}=(0,0)$ extends its stability well beyond the classical value $J_2/J_1=1/6$. Most importantly, the collinear states with ${\bf Q}=(0,2\pi/\sqrt{3})$ (and the two symmetry-related states) are found to be stable in a large regime with intermediate frustration, while at the classical level they are limited to the point $J_2/J_1=0.5$. For large frustration, the $120^\circ$ state is stabilized for finite values of $J_2/J_1$ in both models.