On the Schr\"odinger-Poisson system with steep potential well and indefinite potential

In this paper, we study the following Schr\"odinger-Poisson system: $$ \left\{\aligned&-\Delta u+V_\lambda(x)u+K(x)\phi u=f(x,u)&\quad\text{in }\bbr^3,\\ &-\Delta\phi=K(x)u^2&\quad\text{in }\bbr^3,\\ &(u,\phi)\in\h\times\D,\endaligned\right.\eqno{(\mathcal{SP}_{\lambda})} $$ where $V_\lambda(x)=\lambda a(x)+b(x)$ with a positive parameter $\lambda$, $K(x)\geq0$ and $f(x,t)$ is continuous including the power-type nonlinearity $|u|^{p-2}u$. By applying the method of penalized functions, the existence of one nontrivial solution for such system in the less-studied case $3<p\leq4$ is obtained for $\lambda$ sufficiently large. The concentration behavior of this nontrivial solution for $\lambda\to+\infty$ are also observed. It is worth to point out that some new conditions on the potentials are introduced to obtain this nontrivial solution and the Schr\"odinger operator $-\Delta+V_\lambda(x)$ may be strong indefinite in this paper.