Deligne-Mumford compactification of the real multiplication locus and Teichmueller curves in genus three

In the moduli space M_g of genus g Riemann surfaces, consider the locus RM_O of Riemann surfaces whose Jacobians have real multiplication by the order O in a totally real number field F of degree g. If g = 2 or 3, we compute the closure of RM_O in the Deligne-Mumford compactification of M_g and the closure of the locus of eigenforms over RM_O in the Deligne-Mumford compactification of the moduli space of holomorphic one-forms. For higher genera, we give strong necessary conditions for a stable curve to be in the boundary of RM_O Boundary strata of RM_O are parameterized by configurations of elements of the field F satisfying a strong geometry of numbers type restriction. We apply this computation to give evidence for the conjecture that there are only finitely many algebraically primitive Teichmueller curves in M_3. In particular, we prove that there are only finitely many algebraically primitive Teichmueller curves generated by a one-form having two zeros of order 3 and 1. We also present the results of a computer search for algebraically primitive Teichmueller curves generated by a one-form having a single zero.