Broad emission lines in optical spectra of hot dust-obscured galaxies can contribute significantly to JWST/NIRCam photometry

Selecting the first galaxies at z>7-10 from JWST surveys is complicated by z<6 contaminants with degenerate photometry. For example, strong optical nebular emission lines at z<6 may mimic JWST/NIRCam photometry of z>7-10 Lyman Break Galaxies (LBGs). Dust-obscured 3<z<6 galaxies in particular are potentially important contaminants, and their faint rest-optical spectra have been historically difficult to observe. A lack of optical emission line and continuum measures for 3<z<6 dusty galaxies now makes it difficult to test their expected JWST/NIRCam photometry for degenerate solutions with NIRCam dropouts. Towards this end, we quantify the contribution by strong emission lines to NIRCam photometry in a physically motivated manner by stacking 21 Keck II/NIRES spectra of hot, dust-obscured, massive ($\log\mathrm{M_*/M_\odot}\gtrsim10-11$) and infrared (IR) luminous galaxies at z~1-4. We derive an average spectrum and measure strong narrow (broad) [OIII]5007 and H$\alpha$ features with equivalent widths of $130\pm20$ A ($150\pm50$ A) and $220\pm30$ A ($540\pm80$ A) respectively. These features can increase broadband NIRCam fluxes by factors of 1.2-1.7 (0.2-0.6 mag). Due to significant dust-attenuation ($A_V\sim6$), we find H$\alpha$+[NII] to be significantly brighter than [OIII]+H$\beta$, and therefore find that emission-line dominated contaminants of high-z galaxy searches can only reproduce moderately blue perceived UV continua of $S_\lambda\propto\lambda^\beta$ with $\beta>-1.5$ and z>4. While there are some redshifts (z~3.75) where our stack is more degenerate with the photometry of z>10 LBGs between $\lambda_{rest}\sim0.3-0.8\,\mu$m, redder filter coverage beyond $\lambda_{obs}>3.5\,\mu$m and far-IR/sub-mm follow-up may be useful for breaking the degeneracy and making a crucial separation between two fairly unconstrained populations, dust-obscured galaxies at z~3-6 and LBGs at z>10.