A Lower Bound for Estimating Fr\'echet Means

Fr\'echet means, conceptually appealing, generalize the Euclidean expectation to general metric spaces. We explore how well Fr\'echet means can be estimated from independent and identically distributed samples and uncover a fundamental limitation: In the vicinity of a probability distribution $P$ with nonunique means, independent of sample size, it is not possible to uniformly estimate Fr\'echet means below a precision determined by the diameter of the set of Fr\'echet means of $P$. Implications were previously identified for empirical plug-in estimators as part of the phenomenon \emph{finite sample smeariness}. Our findings thus confirm inevitable statistical challenges in the estimation of Fr\'echet means on metric spaces for which there exist distributions with nonunique means. Illustrating the relevance of our lower bound, examples of extrinsic, intrinsic, Procrustes, diffusion and Wasserstein means showcase either deteriorating constants or slow convergence rates of empirical Fr\'echet means for samples near the regime of nonunique means.