Measurement of Higgs boson self-couplings through $2\rightarrow 3$ vector bosons scattering in future muon colliders

The $2\rightarrow 3$ scattering of longitudinal vector bosons (VBS) has been proven to be a complementary channel to measure the Higgs self-couplings in the Standard Model (SM) and the SM effective field theory (SMEFT) at colliders. We perform the first comprehensive study of all main $2\rightarrow 3$ VBS processes at high-energy muon colliders, especially including background analysis. The main contributing channels turn out to be the scattering of $W^+W^-\rightarrow W^+W^-h$, $ZZh$ and $hhh$. We obtain the constraints on $c_6$ and $c_{\Phi_1}$ which are the Wilson coefficients of the dimension-6 operators relevant for Higgs self-couplings in the SMEFT. With the center-of-mass energies of 10 TeV and 30 TeV, we find the expected sensitivity to the coefficients $c_6/\Lambda^2$ and $c_{\Phi_1}/\Lambda^2$ can reach the level of 0.01 TeV$^{-2}$. The tightest constraints come from the $hhh$ channel, while the constraints from $WWh$ are also comparable. Our results crucially depend on selecting the longitudinal polarizations for the final $W$ and $Z$ bosons. We then study how the sensitivities change by varying the efficiency of tagging longitudinal polarizations, and find that the significance remains consistently high.