Semiclassical ground-state phase diagram and multi-Q phase of a spin-orbit coupled model on triangular lattice

Motivated by recent experiments on the frustrated quantum magnetic compound YbMgGaO4, we study an effective spin model on triangular lattice taking into account the effects of the spin-orbit coupling. We determine the classical ground-state phase diagram of this model, which includes a 120 degree Neel and two collinear antiferromagnetic phases. In the vicinity of the phase boundary between the Neel and collinear phases, we identify three intermediate non-collinear antiferromagnetic phases. In each of them the magnetic moments are ordered at multiple incommensurate wave vector Q values. We further study the effects of quantum fluctuations in this model via a linear spin-wave theory. We find that the spin excitation gap of the non-collinear multi-Q antiferromagnetic state is finite but can be vanishingly small, and this state is unstable to a spin liquid phase under strong quantum fluctuations in some large J_{z+-} regime.