Magnon condensation with finite degeneracy on the triangular lattice

We study the spin 1/2 triangular-lattice $J_1$-$J_2$-$J_3$ antiferromagnet close to the saturation field using the dilute Bose gas theory, where the magnetic structure is determined by the condensation of magnons. We focus on the case of ferromagnetic $J_1$ and antiferromagnetic $J_2,J_3$, that is particularly rich because frustration effects allow the single-magnon energy dispersion to have six-fold degenerate minima at incommensurate momenta. Our calculation also includes an interlayer coupling $J_0$, which covers both antiferromagnetic and ferromagnetic cases including negligibly small regime (two-dimensional case). Besides the spiral and fan phases, we find a new double-$q$ phase (superposition of two modes), dubbed "${\bf Q}_0$-${\bf Q}_1$" (or simply "01") phase, that enjoys a new type of multiferroic character. Certain phase boundaries have a singular $J_0$ dependence for $J_0\to 0$, implying that even a very small interlayer coupling drastically changes the ground state. A mechanism for this singularity is presented. Moreover, in some regions of the parameter space, we show that a dilute gas of magnons can not be stable, and phase separation (corresponding to a magnetization jump) is expected. In the $J_1$-$J_2$ model ($J_3=0$), formation of two-magnon bound states is observed, which can lead to a quadrupolar (spin-nematic) ordered phase. Exact diagonalization analysis is also applied to the search of bound states.