The systematics of strong lens modeling quantified: the effects of constraint selection and redshift information on magnification, mass, and multiple image predictability

Until now, systematic errors in strong gravitational lens modeling have been acknowledged but never been fully quantified. Here, we launch an investigation into the systematics induced by constraint selection. We model the simulated cluster Ares 362 times using random selections of image systems with and without spectroscopic redshifts and quantify the systematics using several diagnostics: image predictability, accuracy of model-predicted redshifts, enclosed mass, and magnification. We find that for models with $>15$ image systems, the image plane rms does not decrease significantly when more systems are added; however the rms values quoted in the literature may be misleading as to the ability of a model to predict new multiple images. The mass is well constrained near the Einstein radius in all cases, and systematic error drops to $<2\%$ for models using $>10$ image systems. Magnification errors are smallest along the straight portions of the critical curve, and the value of the magnification is systematically lower near curved portions. For $>15$ systems, the systematic error on magnification is $\sim2\%$. We report no trend in magnification error with fraction of spectroscopic image systems when selecting constraints at random; however, when using the same selection of constraints, increasing this fraction up to $\sim0.5$ will increase model accuracy. The results suggest that the selection of constraints, rather than quantity alone, determines the accuracy of the magnification. We note that spectroscopic follow-up of at least a few image systems is crucial, as models without any spectroscopic redshifts are inaccurate across all of our diagnostics.