Minimal energy solutions to the fractional Lane-Emden system, I: Existence and singularity formation

This is the first of two papers which study asymptotic behavior of minimal energy solutions to the fractional Lane-Emden system in a smooth bounded domain $\Omega$ \[(-\Delta)^s u = v^p, \quad (-\Delta)^s v = u^q \text{ in } \Omega \quad \text{and} \quad u = v = 0 \text{ on } \pa \Omega \quad \text{for } 0 < s < 1\] under the assumption that the subcritical pair $(p,q)$ approaches to the critical Sobolev hyperbola. If $p = 1$, the above problem is reduced to the subcritical higher-order fractional Lane-Emden equation with the Navier boundary condition \[(-\Delta)^s u = u^{\frac{n+2s}{n-2s}-\ep} \text{ in } \Omega \quad \text{and} \quad u = (-\Delta)^{s \over 2} u = 0 \quad \text{for } 1 < s < 2.\] The main objective of this paper is to deduce the existence of minimal energy solutions, and to examine their (normalized) pointwise limits provided that $\Omega$ is convex. As a by-product of our study, a new approach for the existence of an extremal function for the Hardy-Littlewood-Sobolev inequality is provided.