A$^3$COSMOS: Measuring the cosmic dust-attenuated star formation rate density at $4 < z < 5$

[Abridged] In recent years, conflicting results have provided an uncertain view of the dust-attenuated properties of $z>4$ star-forming galaxies (SFGs). To solve this, we used the deepest data publicly available in COSMOS to build a mass-complete ($>10^{9.5}\,M_{\odot}$) sample of SFGs at $4<z<5$ and measured their dust-attenuated properties by stacking all archival ALMA band 6 and 7 observations available. Combining this information with their rest-frame ultraviolet emission from the COSMOS2020 catalog, we constrained the IRX ($\equiv L_{\rm IR}/L_{\rm UV}$)--$\beta_{\rm UV}$, IRX--$M_\ast$, and SFR--$M_\ast$ relations at $z\sim4.5$. Finally, using these relations and the stellar mass function of SFGs at $z\sim4.5$, we inferred the unattenuated and dust-attenuated SFRD at this epoch. SFGs at $z\sim4.5$ follow an IRX--$\beta_{\rm UV}$ relation that is consistent with that of local starbursts, while they follow a steeper IRX--$M_\ast$ relation than observed locally. The grain properties of dust in these SFGs seems thus similar to those in local starbursts but its mass and geometry result in lower attenuation in low-mass SFGs. SFGs at $z\sim4.5$ lie on a linear SFR--$M_\ast$ relation, whose normalization varies by 0.3 dex, when we exclude or include from our stacks the ALMA primary targets. The cosmic SFRD$(>M_\ast)$ converges at $M_\ast<10^{9}\,M_\odot$ and is dominated by SFGs with $M_\ast\sim10^{9.5-10.5}\,M_\odot$. The fraction of the cosmic SFRD that is attenuated by dust, ${\rm SFRD}_{\rm IR}(>M_\ast)/ {\rm SFRD}(>M_\ast)$, is $90\pm4\%$ for $M_\ast\,=\,10^{10}\,M_\odot$, $68\pm10\%$ for $M_\ast=10^{8.9}\,M_\odot$ (i.e., $0.03\times M^\star$; $M^\star$ being the characteristic stellar mass of SFGs) and this value converges to $60\pm10\%$ for $M_\ast=10^{8}\,M_\odot$. Even at this early epoch, the fraction of the cosmic SFRD that is attenuated by dust remains thus significant.