Nematic excitonic insulator in transition metal dichalcogenide moir\'e heterobilayers

We study the effect of inter-electron Coulomb interactions on the displacement field induced topological phase transition in transition metal dichalcogenide (TMD) moir\'e heterobilayers. We find a nematic excitonic insulator (NEI) phase that breaks the moir\'e superlattice's three-fold rotational symmetry and preempts the topological phase transition in both AA and AB stacked heterobilayers when the interlayer tunneling is weak, or when the Coulomb interaction is not strongly screened. The nematicity originates from the frustration between the nontrivial spatial structure of the interlayer tunneling, which is crucial to the existence of the topological Chern band, and the interlayer coherence induced by the Coulomb interaction that favors uniformity in layer pseudo-spin orientations. We construct a unified effective two-band model that captures the physics near the band inversion and applies to both AA and AB stacked heterobilayers. Within the two-band model, the competition between the NEI phase and the Chern insulator phase can be understood as the switching of the energetic order between the $s$-wave and the $p$-wave excitons upon increasing the interlayer tunneling.