Nonholomorphic $A_4$ modular invariance for fermion masses and mixing in SU(5) GUT

Addressing the fermion flavor structures using modular invariance is a challenging task in the framework of quark-lepton unification. Building on recent applications of modular symmetry in non-supersymmetric models, we propose the first renormalizable $SU(5)$ grand unified theory incorporating level 3 nonholomorphic modular symmetry, $\Gamma_3 \simeq A_4$. This framework constrains Yukawa couplings to polyharmonic Maa{\ss} forms, significantly reducing the number of free parameters while enhancing the predictive power of the models. We present a comprehensive analysis of fermion masses and mixing while tackling key GUT queries such as gauge coupling unification and proton decay. Beyond the minimal $SU(5)$ framework, the Higgs sector incorporates the $45_H$ dimensional Higgs field crucial in differentiating the masses of down quarks and charged leptons, and the fermion sector is extended with three right-handed neutrinos enabling neutrino masses via the type-I seesaw mechanism. We analyze two benchmark models with distinct modular weight and $A_4$ charge assignments. The predicted effective Majorana mass $m_{\beta \beta}$ values align with current neutrinoless double-beta decay experiments, and the effective neutrino mass $m_\beta$ is within the reach of future beta decay searches. The predicted sum of neutrino masses, $\sum m_i$, satisfies the upper bound set by recent cosmological observations. The gauge coupling unification is achieved through a light scalar triplet $\phi_3 \sim (3,3,-1/3)$ and a scalar octet $\phi_5 \sim (8,2,1/2)$ belonging to the $45_H$ Higgs, while proton decay constraints require a highly suppressed Yukawa couplings.