Theory of moir\'e magnetism in twisted bilayer $\alpha$-RuCl$_3$

Twisted heterostructures of van der Waals materials have received much attention for their many remarkable properties. Here, we present a comprehensive theory of the long-range ordered magnetic phases of twisted bilayer $\alpha$-RuCl$_3$ via a combination of first-principles calculations and atomistic simulations. While a monolayer exhibits zigzag antiferromagnetic order with three possible ordering wave vectors, a rich phase diagram is obtained for moir\'e superlattices as a function of interlayer exchange and twist angle. For large twist angles, each layer spontaneously picks a single zigzag ordering wave vector, whereas, for small twist angles, the ground state involves a combination of all three wave vectors in a complex hexagonal domain structure. This multi-domain order minimizes the interlayer energy while enduring the energy cost due to the domain wall formation. Our results indicate that magnetic frustration due to stacking-dependent interlayer exchange in moir\'e superlattices can be used to tune the magnetic ground state and enhance quantum fluctuations in $\alpha$-RuCl$_3$.