Distinct-symmetry spin liquid states and phase diagram of Kitaev-Hubbard model

We report the finding of a series of symmetry distinct spin liquid (SL) states and a rich phase diagram in a half-filled honeycomb lattice Hubbard model with spin-dependent hopping amplitude t'. We first study the magnetic instability of the system and find two antiferromagnetic orders beyond a critical Hubbard U which increases with the ratio t'/t . For $t'$ approaching to $t$, the semimetal (SM) transforms to a U(1) SL and then to the Kitaev Z_2 SL as U increases. In a wide middle range of t'/t, the latter is replaced by a U(1) SL to $SU(2)$ SL transition. The physical properties of the stable SL phases are discussed.