Polynomial values modulo primes on average and sharpness of the larger sieve

This paper is motivated by the following question in sieve theory. Given a subset $X\subset [N]$ and $\alpha\in (0,1/2)$. Suppose that $|X\pmod p|\leq (\alpha+o(1))p$ for every prime $p$. How large can $X$ be? On the one hand, we have the bound $|X|\ll_{\alpha}N^{\alpha}$ from Gallagher's larger sieve. On the other hand, we prove, assuming the truth of an inverse sieve conjecture, that the bound above can be improved (for example, to $|X|\ll_{\alpha}N^{O(\alpha^{2014})}$ for small $\alpha$). The result follows from studying the average size of $|X\pmod p|$ as $p$ varies, when $X=f(\mathbb{Z})\cap [N]$ is the value set of a polynomial $f(x)\in\mathbb{Z}[x]$.