Elliptic flow of charged particles at midrapidity relative to the spectator plane in Pb-Pb and Xe-Xe collisions

Measurements of the elliptic flow coefficient relative to the collision plane defined by the spectator neutrons $v_2${$\Psi_{\rm SP}$} in collisions of Pb ions at center-of-mass energy per nucleon-nucleon pair $\sqrt{s_{\rm NN}}$=2.76 TeV and Xe ions at $\sqrt{s_{\rm NN}}$=5.44 TeV are reported. The results are presented for charged particles produced at midrapidity as a function of centrality and transverse momentum. The ratio between $v_2${$\Psi_{\rm SP}$} and the elliptic flow coefficient relative to the participant plane $v_2${4}, estimated using four-particle correlations, deviates by up to 20% from unity depending on centrality. This observation differs strongly from the magnitude of the corresponding eccentricity ratios predicted by the TRENTo and the elliptic power models of initial state fluctuations that are tuned to describe the participant plane anisotropies. The differences can be interpreted as a decorrelation of the neutron spectator plane and the reaction plane because of fragmentation of the remnants from the colliding nuclei, which points to an incompleteness of current models of initial state fluctuations. A significant transverse momentum dependence of the ratio $v_2${$\Psi_{\rm SP}$}/$v_2${4} is observed in all but the most central collisions, which may help to understand whether momentum anisotropies at low and intermediate transverse momentum have a common origin in initial state fluctuations. The ratios of $v_2${$\Psi_{\rm SP}$} and $v_2${4} to the corresponding initial state eccentricities for Xe-Xe and Pb-Pb collisions at similar initial entropy density show a difference of $(7.0 \pm 0.9)$% with an additional variation of +1.8% when including RHIC data in the TRENTo parameter extraction. These observations provide new experimental constraints for viscous effects in the hydrodynamic modeling of the expanding quark-gluon plasma.