Branching laws for minimal holomorphic representations

In this paper we study the branching law for the restriction from $SU(n,m)$ to $SO(n,m)$ of the minimal representation in the analytic continuation of the scalar holomorphic discrete series. We identify the the group decomposition with the spectral decomposition of the action of the Casimir operator on the subspace of $S(O(n) \times O(m))$-invariants. The Plancherel measure of the decomposition defines an $L^2$-space of functions, for which certain continuous dual Hahn polynomials furnish an orthonormal basis. It turns out that the measure has point masses precisely when $n-m>2$. Under these conditions we construct an irreducible representation of $SO(n,m)$, identify it with a parabolically induced representation, and construct a unitary embedding into the representation space for the minimal representation of $SU(n,m)$.