Asymptotic behavior of nonlocal $p$-Rayleigh quotients

Let $N\geq 1$, $s,k\in(0,1)$, $p\in(1,\infty)$. Let $t>1$, open bounded set $\Omega\subset\mathbb R^N$, $R$ be the radius of $\Omega$. Let $B_{tR}(\Omega)$ be the ball containing $\Omega$ with radius $tR$ and with the same center as $\Omega$. In this article we study the asymptotic behavior of the first $(s,p)$-eigenvalue and corresponding first $(s,p)$-eigenfunctions during the approximation $k\rightarrow s$. We show that there exhibits a different phenomenon between the two directions of discontinuity of $k\rightarrow s^-$ and continuity of $k\rightarrow s^+$, which can be triggered by behaviors of eigenfunctions on the boundary points bearing the positive Besov Capacity. And this difference prompts us to study the boundary behavior of operators $(-\Delta_p)^s$ on the irregular boundary points. We also characterize some equivalent forms of the continuity case when $k\rightarrow s^-$. In the end, we construct a counterexample for the discontinuity case during $k\rightarrow s^-$ based on the positivity of Besov capacity of Cantor set and the fine decay estimates up to the regular boundary points, used by P. Lindqvist and O. Martio. The proof works by reducing $\widetilde W^{s,p}_0(\Omega)$ to the so-called Relative-nonlocal spaces $\widetilde W^{s,p}_{0,tR}(\Omega)$ introduced here, which is equivalent to $\widetilde W^{s,p}_0(\Omega)$, where $\widetilde W^{s,p}_0(\Omega)$ is defined as the completion of $C^\infty_0(\Omega)$ under the Gagliardo semi-norm $W^{s,p}(\mathbb R^N)$, and $\widetilde W^{s,p}_{0,tR}(\Omega)$ defined as the completion of $C^\infty_0(\Omega)$ under the Gagliardo semi-norm $W^{s,p}(B_{tR}(\Omega))$. As a partial result, we established the Homemorphism of the operator $(-\Delta_p)^s$ between $\widetilde W^{s,p}_0(\Omega)$ and its dual space $\widetilde W^{-s,p^\prime}(\Omega)$, where $1/p+1/p^\prime=1$.