Semilinear elliptic equations with the pseudo-relativistic operator on a bounded domain

We study the Dirichlet problem for the semilinear equations involving the pseudo-relativistic operator on a bounded domain, (\sqrt{-\Delta + m^2} - m)u =|u|^{p-1}u \quad \textrm{in}~\Omega, with the Dirichlet boundary condition $u=0$ on $\partial \Omega$. Here, $p \in (1,\infty)$ and the operator $(\sqrt{-\Delta + m^2} - m)$ is defined in terms of spectral decomposition. In this paper, we investigate existence and nonexistence of a nontrivial solution, depending on the choice of $p$, $m$ and $\Omega$. Precisely, we show that $(i)$ if $p$ is not $H^1$ subcritical ($p \geq \frac{n+2}{n-2}$) and $\Omega$ is star-shaped, the equation has no nontrivial solution for all $m > 0$; $(ii)$ if $p$ is not $H^{1/2}$ supercritical ($1 <p \leq \frac{n+1}{n-1}$), then there exists a least energy solution for all $m>0$ and any bounded domain $\Omega$; $(iii)$ finally, in the intermediate range ($\frac{n+1}{n-1}<p<\frac{n+2}{n-2}$), the problem has a nontrivial solution, provided that $m$ is sufficiently large and the problem -\Delta u = |u|^{p-1}u \quad \textrm{in}~\Omega, \qquad u =0\quad \textrm{on}~\partial \Omega admits a non-degenerate nontrivial solution, for example, when $\Omega$ is a ball or an annulus.