Correlation effects on topological insulator

The strong correlation effects on topological insulator are studied in a two-sublattice system with an onsite single-particle energy difference $\Delta$ between two sublattices. At $\Delta=0$, increasing the onsite interaction strength $U$ drives the transition from the quantum spin Hall insulating state to the non-topological antiferromagnetic Mott-insulating (AFMI) state. When $\Delta$ is larger than a certain value, a topologically trivial band insulator or AFMI at small values of $U$ may change into a quantum anomalous Hall state with antiferromagnetic ordering at intermediate values of $U$. Further increasing $U$ drives the system back into the topologically trivial state of AFMI. The corresponding phenomena is observable in the solid state and cold atom systems. We also propose a scheme to realize and detect these effects in cold atom systems.