Competing topological and Kondo insulator phases on a honeycomb lattice

We investigate the competition between the spin-orbit interaction of itinerant electrons and their Kondo coupling with local moments densely distributed on the honeycomb lattice. We find that the model at half-filling displays a quantum phase transition between topological and Kondo insulators at a nonzero Kondo coupling. In the Kondo-screened case, tuning the electron concentration can lead to a new topological insulator phase. The results suggest that the heavy-fermion phase diagram contains a new regime with a competition among topological, Kondo-coherent and magnetic states, and that the regime may be especially relevant to Kondo lattice systems with $5d$-conduction electrons. Finally, we discuss the implications of our results in the context of the recent experiments on SmB$_6$ implicating the surface states of a topological insulator, as well as the existing experiments on the phase transitions in SmB$_6$ under pressure and in CeNiSn under chemical pressure.