Transverse-mode coupling and diffraction loss in tunable Fabry-P\'erot microcavities

We report on measurements and modeling of the mode structure of tunable Fabry-P\'erot optical microcavities with imperfect mirrors. We find that non-spherical mirror shape and finite mirror size lead to loss, mode deformation, and shifted resonance frequencies at particular mirror separations. For small mirror diameters, the useful cavity length is limited to values significantly below the expected stability range. We explain the observations by resonant coupling between different transverse modes of the cavity and mode-dependent diffraction loss. A model based on resonant state expansion that takes into account the measured mirror profile can reproduce the measurements and identify the parameter regime where detrimental effects of mode mixing are avoided.