Antiferroquadrupolar order and rotational symmetry breaking in a generalized bilinear-biquadratic model on a square lattice

The magnetic and nematic properties of the iron chalcogenides have recently been the subject of intense interest. Motivated by the proposed antiferroquadrupolar and Ising-nematic orders for the bulk FeSe, we study the phase diagram of an $S=1$ generalized bilinear-biquadratic model with multi-neighbor interactions. We find a large parameter regime for a ($\pi$,0) antiferroquadrupolar phase, showing how quantum fluctuations stabilize it by lifting an infinite degeneracy of certain semiclassical states. Evidence for this C$_4$-symmetry-breaking quadrupolar phase is also provided by an unbiased density matrix renormalization group analysis. We discuss the implications of our results for FeSe and related iron-based superconductors.