Effect of surface hybridization on RKKY coupling in ferromagnet/topological insulator/ferromagnet trilayer system

We theoretically investigate the RKKY exchange coupling between two ferromagnets (FM) separated by a thin topological insulator film (TI). We find an unusual dependence of the RKKY exchange coupling on the TI thickness ($t_{TI}$). First, when $t_{TI}$ decreases, the coupling amplitude increases at first and reaches its maximum value at some critical thickness, below which the amplitude turns to diminish. This trend is attributed to the hybridization between surfaces of the TI film, which opens a gap below critical thickness and thus turns the surfaces into insulating state from semi-metal state. In insulating phase, diamagnetism induced by the gap-opening compensates paramagnetism of Dirac state, resulting in a diminishing magnetic susceptibility and RKKY coupling. For typical parameters, the critical thickness in Bi2Se3 thin film is estimated to be in the range of 3-5 nm.