Emergence of Quantum Nonmagnetic Insulating Phase in Spin-Orbit Coupled Square Lattices

We investigate the metal-insulator transition (MIT) and phase diagram of the half-filled Fermi Hubbard model with Rashba-type spin-orbit coupling (SOC) on a square optical lattice. The interplay between the atomic interactions and SOC results in distinctive features of the MIT. Significantly, in addition to the diverse spin ordered phases, a nonmagnetic insulating phase emerges in a considerably large regime of parameters near the Mott transition. This phase has a finite single-particle gap but vanishing magnetization and spin correlation exhibits a power-law scaling, suggesting a potential algebraic spin-liquid ground state. These results are confirmed by the non-perturbative cluster dynamical mean-field theory.