Helical Majorana fermions in d_(x²-y²) + i d_(xy)-wave topological superconductivity of doped correlated quantum spin Hall insulators

Large Hubbard U limit of the Kane-Mele model on a zigzag ribbon of honeycomb lattice near half-filling is studied via a renormalized mean-field theory. The ground state exhibits time-reversal symmetry (TRS) breaking d + i d'-wave superconductivity. At large spin-orbit coupling, the Z2 phase with non-trivial spin Chern number in the pure Kane-Mele model is persistent into the TRS broken state (called spin-Chern phase), and has two pairs of counter-propagating helical Majorana modes at the edges. As the spin-orbit coupling is reduced, the system undergoes a topological quantum phase transition from the spin-Chern to chiral superconducting states. Possible relevance of our results to adatom-doped graphene and irridate compounds is discussed.