Detecting dense-matter phase transition signatures in neutron star mass-radius measurements as data anomalies using normalising flows

Observations of neutron stars may be used to study aspects of extremely dense matter, specifically a possibility of phase transitions to exotic states, such as de-confined quarks. We present a novel data analysis method for detecting signatures of dense-matter phase transitions in sets of mass-radius measurements, and study its sensitivity with respect to the size of observational errors and the number of observations. The method is based on machine learning anomaly detection coupled with normalizing flows technique: the algorithm trained on samples of astrophysical observations featuring no phase transition signatures interprets a phase transition sample as an ''anomaly''. For the sake of this study, we focus on dense-matter equations of state leading to detached branches of mass-radius sequences (strong phase transitions), use an astrophysically-informed neutron-star mass function, and various magnitudes of observational errors and sample sizes. The method is shown to reliably detect cases of mass-radius relations with phase transition signatures, while increasing its sensitivity with decreasing measurement errors and increasing number of observations. We discuss marginal cases, when the phase transition mass is located near the edges of the mass function range. Evaluated on the current state-of-art selection of real measurements of electromagnetic and gravitational-wave observations, the method gives inconclusive results, which we interpret as due to small available sample size, large observational errors and complex systematics.