Enhanced observable estimation through classical optimization of informationally over-complete measurement data -- beyond classical shadows

In recent years, informationally complete measurements have attracted considerable attention, especially in the context of classical shadows. In the particular case of informationally over-complete measurements, for which the number of possible outcomes exceeds the dimension of the space of linear operators in Hilbert space, the dual POVM operators used to interpret the measurement outcomes are not uniquely defined. In this work, we propose a method to optimize the dual operators after the measurements have been carried out in order to produce sharper, unbiased estimations of observables of interest. We discuss how this procedure can produce zero-variance estimations in cases where the classical shadows formalism, which relies on so-called canonical duals, incurs exponentially large measurement overheads. We also analyze the algorithm in the context of quantum simulation with randomized Pauli measurements, and show that it can significantly reduce statistical errors with respect to canonical duals on multiple observable estimations.