Improving Galaxy Cluster Selection with the Outskirt Stellar Mass of Galaxies

The number density and redshift evolution of optically selected galaxy clusters offer an independent measurement of the amplitude of matter fluctuations, $S_8$. However, recent results have shown that clusters chosen by the redMaPPer algorithm show richness-dependent biases that affect the weak lensing signals and number densities of clusters, increasing uncertainty in the cluster mass calibration and reducing their constraining power. In this work, we evaluate an alternative cluster proxy, outskirt stellar mass, $M_{\textrm{out}}$, defined as the total stellar mass within a $[50,100]$ kpc envelope centered on a massive galaxy. This proxy exhibits scatter comparable to redMaPPer richness, $\lambda$, but is less likely to be subject to projection effects. We compare the Dark Energy Survey Year 3 redMaPPer cluster catalog with a $M_{\textrm{out}}$ selected cluster sample from the Hyper-Suprime Camera survey. We use weak lensing measurements to quantify and compare the scatter of $M_{\textrm{out}}$ and $\lambda$ with halo mass. Our results show $M_{\textrm{out}}$ has a scatter consistent with $\lambda$, with a similar halo mass dependence, and that both proxies contain unique information about the underlying halo mass. We find $\lambda$-selected samples introduce features into the measured $\Delta \Sigma$ signal that are not well fit by a log-normal scatter only model, absent in $M_{\textrm{out}}$ selected samples. Our findings suggest that $M_{\textrm{out}}$ offers an alternative for cluster selection with more easily calibrated selection biases, at least at the generally lower richnesses probed here. Combining both proxies may yield a mass proxy with a lower scatter and more tractable selection biases, enabling the use of lower mass clusters in cosmology. Finally, we find the scatter and slope in the $\lambda-M_{\textrm{out}}$ scaling relation to be $0.49 \pm 0.02$ and $0.38 \pm 0.09$.