Ground state phase diagram and the exotic phases in the spin-1/2 square lattice J1-J2-Jx model

The intricate interplay between frustration and spin chirality has the potential to give rise to unprecedented phases in frustrated quantum magnets. We examine the ground state phase diagram of the spin-1/2 square lattice J1-J2-Jx model by employing critical level crossings and ground state fidelity susceptibility (FS) using exact diagonalization (ED) with full lattice symmetries. Our analysis reveals the evolution of highly symmetric energy levels as a function of J2 at fixed Jx. During a magnetic to non-magnetic phase transition, the precise identification of the phase boundary is achieved through critical level crossings between the gapless excitation of a magnetic phase and the quasi-degenerate ground state of a non-magnet phase. Conversely, a direct transition between two non-magnetic phases is characterized by a FS peak accompanied by an avoided ground state level crossing, serving as a distinctive signal. Within a substantial range of Jx, we identify an anticipated chiral spin liquid (CSL) state and an adjacent nematic spin liquid (NSL) phase with a degeneracy of two on a cylinder. These two phases are demarcated by a nearly vertical boundary line at J2 = 0.65. This critical line terminates at the lower boundary of a magnetic ordered chiral spin solid (CSS) phase. We validate the topological nature of the CSL using the modular S matrix of the minimum entangled states (MES) on a torus, along with the entanglement spectra (ES) of even and odd sectors on a cylinder, employing an SU(2)-symmetric density matrix renormalization group (DMRG) method. Furthermore, we delve into a comprehensive discussion on the nature of the NSL, exploring aspects such as ground state degeneracy, the local bond energy landscape, and the singlet and triplet gaps on various tori. These analysis provide substantial evidence supporting the nematic nature of the NSL.