Exploiting non-orthogonal eigenmodes in a non-Hermitian optical system to realize sub-100 MHz magneto-optical filters

Non-Hermitian physics is responsible for many of the counter-intuitive effects observed in optics research opening up new possibilities in sensing, polarization control and measurement. A hallmark of non-Hermitian matrices is the possibility of non-orthogonal eigenvectors resulting in coupling between modes. The advantages of propagation mode coupling have been little explored in magneto-optical filters and other devices based on birefringence. Magneto-optical filters select for ultra-narrow transmission regions by passing light through an atomic medium in the presence of a magnetic field. Passive filter designs have traditionally been limited by Doppler broadening of thermal vapors. Even for filter designs incorporating a pump laser, transmissions are typically less than 15\% for sub-Doppler features. Here we exploit our understanding of non-Hermitian physics to induce non-orthogonal propagation modes in a vapor and realize better magneto-optical filters. We construct two new filter designs with ENBWs and maximum transmissions of 181~MHz, 42\% and 140~MHz, 17\% which are the highest figure of merit and first sub-100~MHz FWHM passive filters recorded respectively. This work opens up a range of new filter applications including metrological devices for use outside a lab setting and commends filtering as a new candidate for deeper exploration of non-Hermitian physics such as exceptional points of degeneracy.