Non-Clifford diagonalization for measurement shot reduction in quantum expectation value estimation

Estimating expectation values on near-term quantum computers often requires a prohibitively large number of measurements. One widely-used strategy to mitigate this problem has been to partition an operator's Pauli terms into sets of mutually commuting operators. Here, we introduce a method that relaxes this constraint of commutativity, instead allowing for entirely arbitrary terms to be grouped together, save a locality constraint. The key idea is that we decompose the operator into arbitrary tensor products with bounded tensor size, ignoring Pauli commuting relations. This method -- named $k$-NoCliD ($k$-local non-Clifford diagonalization) -- allows one to measure in far fewer bases in most cases, often (though not always) at the cost of increasing the circuit depth. We introduce several partitioning algorithms tailored to different Hamiltonian classes. For electronic structure, we numerically demonstrate the existence of threshold values of $k$ for which $k$-NoCliD leads to the lowest shot counts, though we leave improved partitioning algorithms to future work. We focus primarily on three Hamiltonian classes -- molecular vibrational structure, Fermi-Hubbard, and Bose-Hubbard -- and show that $k$-NoCliD reduces the number of circuit shots, often by a very large margin, and often even for $k$ as small as 2.