Intertwined nematic orders in a frustrated ferromagnet

We investigate the quantum phases of the frustrated spin-$\frac{1}{2}$ $J_1$-$J_2$-$J_3$ Heisenberg model on the square lattice with ferromagnetic $J_1$ and antiferromagnetic $J_2$ and $J_3$ interactions. Using the pseudo-fermion functional renormalization group technique, we find an intermediate paramagnetic phase located between classically ordered ferromagnetic, stripy antiferromagnetic, and incommensurate spiral phases. We observe that quantum fluctuations lead to significant shifts of the spiral pitch angles compared to the classical limit. By computing the response of the system with respect to various spin rotation and lattice symmetry-breaking perturbations, we identify a complex interplay between different nematic spin states in the paramagnetic phase. While retaining time-reversal invariance, these phases either break spin-rotation symmetry, lattice-rotation symmetry, or a combination of both. We therefore propose the $J_1$-$J_2$-$J_3$ Heisenberg model on the square lattice as a paradigmatic example where different intimately connected types of nematic orders emerge in the same model.