Overlapped groupings for quantum energy estimation: Maximal variance reduction and deterministic algorithms for reducing variance
Pith reviewed 2026-05-10 17:51 UTC · model grok-4.3
The pith
Overlapped grouping of Hamiltonian terms reduces variance in quantum energy estimates linearly with the number of terms.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that overlapped grouping for energy estimation achieves a maximal variance reduction linear in the number of Hamiltonian terms. The repacking algorithm converts disjoint groups into overlapped groups and is shown to reduce variance iteratively. Simulations on systems with up to 575,000 terms demonstrate that the variance reduction relative to disjoint grouping increases linearly with problem size.
What carries the argument
The repacking procedure, which redistributes coefficients of Hamiltonian terms across multiple compatible groups to minimize the total variance of the energy estimator.
If this is right
- The number of measurements required for a fixed precision in energy estimation decreases as the Hamiltonian grows larger.
- Existing disjoint grouping schemes can be systematically improved by a deterministic procedure without introducing randomness.
- The advantage of overlapped grouping widens with problem size, making it more attractive precisely when the Hamiltonian contains many terms.
- The method applies directly to any Hamiltonian that can be measured via Pauli strings or other compatible operator sets.
Where Pith is reading between the lines
- The linear scaling may extend the practical reach of variational quantum eigensolvers to molecules or materials whose Hamiltonians contain millions of terms.
- If the mild assumptions hold for typical molecular Hamiltonians, the approach could be combined with other measurement-reduction techniques such as classical shadows.
- A natural next test would be to measure whether the same repacking logic improves variance in expectation values of observables other than the energy.
Load-bearing premise
The repacking procedure reduces variance iteratively only under mild assumptions on the initial groups and the structure of the Hamiltonian.
What would settle it
A concrete Hamiltonian for which repeated application of the repacking algorithm fails to decrease variance, or for which the observed reduction remains sub-linear even as the number of terms grows into the hundreds of thousands.
Figures
read the original abstract
Grouping-based measurement strategies are widely used to reduce measurement complexity in near-term quantum algorithms. While these schemes have typically produced disjoint groups, recently this has been relaxed in what is known as overlapped grouping or coefficient splitting where operators may appear in more than one compatible group. In recent work, it has been numerically shown that this strategy can reduce the variance of energy estimates on small benchmark problems, motivating both the application and further analysis of the method. Here we prove that overlapped grouping for energy estimation can lead to a maximal variance reduction that is linear in the number of Hamiltonian terms. We introduce a new algorithm which we call repacking to transform existing groups into overlapped groups, and we show this repacking procedure iteratively reduces variance under mild assumptions. We also perform numerical simulations with Hamiltonians up to $44$ qubits and $575 \cdot 10^{3}$ terms, assessing overlapped grouping at scale on problems of practical importance. Our numerics show that the variance reduction relative to state-of-the-art (disjoint) grouping increases linearly with the problem size, suggesting that overlapped grouping methods can be a powerful strategy for quantum energy estimation at the scale of Megaquop computers and beyond.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proves that overlapped grouping (via coefficient splitting) for quantum energy estimation can achieve a maximal variance reduction that scales linearly with the number of Hamiltonian terms. It introduces a deterministic 'repacking' algorithm to convert disjoint groups into overlapped ones and shows that this procedure iteratively reduces variance under mild assumptions. Large-scale numerics on Hamiltonians up to 44 qubits and 575k terms demonstrate that the variance reduction relative to state-of-the-art disjoint grouping grows linearly with problem size.
Significance. If the proof and its assumptions hold generally, the linear scaling offers a substantial improvement in measurement overhead for near-term variational quantum algorithms, with clear relevance to scaling toward larger systems. The deterministic algorithm and extensive numerics on practically relevant instances are strengths that enhance the result's applicability.
major comments (2)
- [§3] §3 (repacking algorithm and variance-reduction proof): The central claim of linear (in number of terms) maximal variance reduction is established by showing iterative improvement under 'mild assumptions,' but these assumptions are not explicitly enumerated or their robustness tested against common Hamiltonian features such as mixed-sign coefficients or non-trivial commuting structures; this is load-bearing because early termination of the iteration would prevent the claimed linear factor.
- [§5] §5 (numerical results, e.g., the scaling plots): The observed linear growth is reported for specific instances, but the manuscript does not compare the achieved variance reduction against the theoretical upper bound derived in the proof, leaving open whether the repacking procedure attains the claimed maximum or saturates earlier on the tested Hamiltonians.
minor comments (2)
- [§2-3] Notation for the overlapped groups and the variance expression could be unified between the proof and the algorithm pseudocode to avoid reader confusion.
- [§5] Figure captions in the numerics section should explicitly state the number of terms and qubits for each data point to facilitate direct comparison with the linear scaling claim.
Simulated Author's Rebuttal
We thank the referee for their careful reading, positive assessment of the significance of our results, and constructive comments. We address each major comment below and outline revisions that will strengthen the manuscript.
read point-by-point responses
-
Referee: [§3] §3 (repacking algorithm and variance-reduction proof): The central claim of linear (in number of terms) maximal variance reduction is established by showing iterative improvement under 'mild assumptions,' but these assumptions are not explicitly enumerated or their robustness tested against common Hamiltonian features such as mixed-sign coefficients or non-trivial commuting structures; this is load-bearing because early termination of the iteration would prevent the claimed linear factor.
Authors: We agree that the assumptions should be stated explicitly for clarity and to allow readers to assess their generality. In the revised manuscript we will add an enumerated list of the mild assumptions directly in §3, including the precise conditions under which the iterative repacking is guaranteed to continue. The proof itself is formulated for general coefficients (including mixed signs) and does not rely on all-positive coefficients or trivial commutation relations; we will add a short paragraph clarifying this generality and noting that the assumptions are satisfied by the molecular and spin Hamiltonians used in our numerics. We will also include a brief robustness discussion, supported by additional small-scale tests on Hamiltonians with varied sign patterns and commuting structures, to confirm that early termination does not occur for the problem classes of interest. revision: yes
-
Referee: [§5] §5 (numerical results, e.g., the scaling plots): The observed linear growth is reported for specific instances, but the manuscript does not compare the achieved variance reduction against the theoretical upper bound derived in the proof, leaving open whether the repacking procedure attains the claimed maximum or saturates earlier on the tested Hamiltonians.
Authors: We acknowledge that an explicit comparison to the theoretical upper bound would be valuable. For the largest instances (hundreds of thousands of terms) an exact computation of the bound is intractable, as it requires solving a combinatorial optimization problem over all possible overlapped groupings. In the revision we will add a dedicated paragraph in §5 that (i) recalls the theoretical linear upper bound, (ii) explains the computational intractability for large systems, and (iii) presents new comparisons on smaller Hamiltonians (where the bound is computable via exhaustive or integer-programming methods) showing that repacking reaches reductions within a few percent of the maximum. These smaller-scale results, together with the observed linear scaling on the large instances, support that the algorithm attains the claimed maximal reduction in practice. revision: partial
Circularity Check
No circularity: proof and algorithm are independent of inputs
full rationale
The paper derives the linear variance reduction claim via a mathematical proof and introduces a deterministic repacking algorithm whose iterative improvement is shown under explicitly stated mild assumptions. These steps are first-principles constructions that do not reduce to self-definitions, fitted parameters renamed as predictions, or load-bearing self-citations. Numerical experiments on large Hamiltonians serve as validation rather than the derivation itself. The central result remains self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Post-hoc repacking Algorithm 1:Post-hoc repacking 1InitializeG ′[j] ←G [j] for allj= 1, . . . , m; 2forj= 1, . . . , mdo 3LetU j be the diagonalizing unitary forG[j]; 4foreachP i ∈Hdo 5P ′ i ←U j PiU † j; 6ifP ′ i contains onlyZandIthen 7G ′[j] ←G ′[j] ∪ {Pi}; 8OutputR={G ′[1], . . . , G′[m]}; The post-hoc repacking scheme is motivated by basis- first mea...
-
[2]
Ad-hoc repacking Algorithm 2:Ad-hoc repacking 1InitializeG ′[j] ←G [j] for allj= 1, . . . , m; 2foreach Pauli operatorP i do 3µ i ← {j:P i ∈G ′[j] } ; 4whilethere exists(P i, j)such thatP i /∈G′[j] andP i commutes with all operators inG ′[j] do 5SelectP i maximizingC(P i) =c 2 i /µi; 6InsertP i into the first compatible groupG′[j] such that Pi /∈G′[j]; 7G...
-
[3]
Throughout this section, we refer to Fig
Maximal variance reduction for overlapped grouping: Proof of Theorem 1 (Restated as Theorem 4) In this section we prove Theorem 1 on the maxi- mal variance reduction possible for overlapped group- ing, which we restate for convenience below. Throughout this section, we refer to Fig. 1 to illustrate the family of Hamiltonians which admit this maximal varia...
-
[4]
The shot batches across different groups are inde- pendent
-
[5]
For any groupG[j], the estimators of distinct Pauli terms measured in that group have zero covariance, i.e., Cov ⟨Pi⟩(j), ⟨Pk⟩(j) = 0for all distinctP i, Pk ∈G [j]. Then the minimum-variance unbiased energy estima- tor (17) is attained by the heuristic estimator (18). Proof.Under assumptions (1)–(2), all covariance terms between distinct Pauli observables...
-
[6]
Alloperatorswhoselabelscontainanuppercaselet- ter mutually commute
-
[7]
A lowercase-only operatorℓcommutes with exactly those operators whose label ends in the same lower- case letterℓ, and anticommutes with all operators whose lowercase label differs. Refer to Fig. 1 for an illustration of the Hamiltonian (commutativity graph). Choose coefficientsc i such that|c i|= 1 +ϵ=O(1) for all operators. Here−1/N≤ϵ≤1/Nis a small pertu...
-
[8]
Deterministic algorithms for reducing variance: Proof of Theorem 2 (Restated as Theorem 5) The primary goal of this section is to prove Theorem 2, which we restate for convenience below. Intuitively, this result says that overlapped groupings found by repacking always have smaller variance than the disjoint groupings from which they arise, assuming zero c...
-
[9]
Love, Alán Aspuru- Guzik, and Jeremy L
Alberto Peruzzo, Jarrod McClean, Peter Shadbolt, Man- Hong Yung, Xiao-Qi Zhou, Peter J. Love, Alán Aspuru- Guzik, and Jeremy L. O’Brien. A variational eigenvalue solver on a photonic quantum processor.Nature Com- munications, 5(1):4213, July 2014
work page 2014
-
[10]
Jarrod R McClean, Jonathan Romero, Ryan Babbush, andAlánAspuru-Guzik. Thetheoryofvariationalhybrid quantum-classical algorithms.New Journal of Physics, 18(2):023023, February 2016
work page 2016
-
[11]
Dave Wecker, Matthew B. Hastings, and Matthias Troyer. Progress towards practical quantum variational algorithms.Physical Review A, 92(4):042303, October 2015
work page 2015
-
[12]
Vladyslav Verteletskyi, Tzu-Ching Yen, and Artur F. Iz- maylov. Measurement optimization in the variational quantum eigensolver using a minimum clique cover.The Journal of Chemical Physics, 152(12):124114, March 2020
work page 2020
-
[13]
Kirby, Shu Fay Ung, Akimasa Miyake, and Peter J
Andrew Zhao, Andrew Tranter, William M. Kirby, Shu Fay Ung, Akimasa Miyake, and Peter J. Love. Mea- surement reduction in variational quantum algorithms. Physical Review A, 101(6):062322, June 2020
work page 2020
-
[14]
Ophelia Crawford, Barnaby van Straaten, Daochen Wang, Thomas Parks, Earl Campbell, and Stephen Brierley. Efficient quantum measurement of Pauli oper- ators in the presence of finite sampling error.Quantum, 5:385, January 2021
work page 2021
-
[15]
Hsin-Yuan Huang, Richard Kueng, and John Preskill. Predicting many properties of a quantum system from very few measurements.Nature Physics, 16(10):1050– 1057, October 2020
work page 2020
-
[16]
Charles Hadfield, Sergey Bravyi, Rudy Raymond, and Antonio Mezzacapo. Measurements of Quantum Hamil- tonians with Locally-Biased Classical Shadows.Commu- nications in Mathematical Physics, 391(3):951–967, May 2022
work page 2022
-
[17]
Hsin-YuanHuang, RichardKueng, andJohnPreskill. Ef- ficient Estimation of Pauli Observables by Derandomiza- tion.Physical Review Letters, 127(3):030503, July 2021
work page 2021
-
[18]
Xavier Bonet-Monroig, Ryan Babbush, and Thomas E. O’Brien. Nearlyoptimalmeasurementschedulingforpar- tial tomography of quantum states.Physical Review X, 10(3):031064, 2020
work page 2020
-
[19]
Rossi, Boris Sokolov, Francesco Tacchino, Panagiotis Kl
Guillermo García-Pérez, Matteo A.C. Rossi, Boris Sokolov, Francesco Tacchino, Panagiotis Kl. Barkoutsos, Guglielmo Mazzola, Ivano Tavernelli, and Sabrina Man- iscalco. Learning to measure: Adaptive informationally complete generalized measurements for quantum algo- rithms.PRX Quantum, 2(4):040342, November 2021
work page 2021
-
[20]
Giacomo Torlai, Guglielmo Mazzola, Giuseppe Carleo, and Antonio Mezzacapo. Precise measurement of quan- tum observables with neural-network estimators.Physi- cal Review Research, 2(2):022060, 2020
work page 2020
-
[21]
Adaptive quantum state tomography with neural networks.npj Quantum Information, 7(1):105, 2021
YihuiQuek, StanislavFort, andHuiKhoonNg. Adaptive quantum state tomography with neural networks.npj Quantum Information, 7(1):105, 2021
work page 2021
-
[22]
Alistair W. R. Smith, Johnnie Gray, and M. S. Kim. Efficient quantum state sample tomography with basis- dependent neural networks.PRX Quantum, 2(2):020348, 2021
work page 2021
-
[23]
William J. Huggins, Jarrod R. McClean, Nicholas C. Ru- bin, Zhang Jiang, Nathan Wiebe, K. Birgitta Whaley, and Ryan Babbush. Efficient and noise resilient mea- surements for quantum chemistry on near-term quantum computers.npj Quantum Information, 7(1):23, 2021
work page 2021
- [24]
-
[25]
Bujiao Wu, Jinzhao Sun, Qi Huang, and Xiao Yuan. Overlapped grouping measurement: A unified framework for measuring quantum states.Quantum, 7:896, January 2023. 16
work page 2023
-
[26]
Tzu-Ching Yen, Aadithya Ganeshram, and Artur F. Iz- maylov. Deterministic improvements of quantum mea- surements with grouping of compatible operators, non- local transformations, and covariance estimates.npj Quantum Information, 9(1):14, February 2023
work page 2023
-
[27]
Alexander Gresch and Martin Kliesch. Guaranteed effi- cient energy estimation of quantum many-body Hamilto- nians using ShadowGrouping.Nature Communications, 16(1):689, January 2025
work page 2025
-
[28]
Seonghoon Choi, Tzu-Ching Yen, and Artur F Iz- maylov. Improving quantum measurements by introduc- ing “ghost” pauli products.Journal of Chemical Theory and Computation, 18(12):7394–7402, 2022
work page 2022
-
[29]
Beyond nisq: The megaquop machine
John Preskill. Beyond nisq: The megaquop machine. ACM Transactions on Quantum Computing, 6(3):1–7,
- [30]
-
[31]
Ben DalFavero, Rahul Sarkar, Jeremiah Rowland, Daan Camps, Nicolas P. D. Sawaya, and Ryan LaRose. Measurement reduction for expectation val- ues via fine-grained commutativity.Physical Review A, 112(5):052407, November 2025
work page 2025
-
[32]
Michael R. Garey and David S. Johnson.Comput- ers and Intractability; A Guide to the Theory of NP- Completeness. W. H. Freeman & Co., USA, October 1990
work page 1990
-
[33]
Nicholas C Rubin, Ryan Babbush, and Jarrod Mc- Clean. Application of fermionic marginal constraints to hybrid quantum algorithms.New Journal of Physics, 20(5):053020, May 2018
work page 2018
-
[34]
Jena, Priyanka Mukhopad- hyay, JanF.Haase, FelixLeditzky, andLucaDellantonio
Ariel Shlosberg, Andrew J. Jena, Priyanka Mukhopad- hyay, JanF.Haase, FelixLeditzky, andLucaDellantonio. Adaptive estimation of quantum observables.Quantum, 7:906, January 2023
work page 2023
-
[35]
Improved simulation of stabilizer circuits.Physical Review A, 70(5):052328, November 2004
Scott Aaronson and Daniel Gottesman. Improved simulation of stabilizer circuits.Physical Review A, 70(5):052328, November 2004
work page 2004
-
[36]
Sergey Bravyi, Ruslan Shaydulin, Shaohan Hu, and Dmitri Maslov. Clifford Circuit Optimization with Tem- plates and Symbolic Pauli Gates.Quantum, 5:580, November 2021
work page 2021
-
[37]
Nicolas PD Sawaya, Daniel Marti-Dafcik, Yang Ho, Daniel P. Tabor, David E. Bernal Neira, Alicia B. Mag- ann, Shavindra Premaratne, Pradeep Dubey, Anne Mat- suura, Nathan Bishop, Wibe A. de Jong, Simon Ben- jamin, Ojas Parekh, Norm Tubman, Katherine Klymko, andDaanCamps. HamLib: AlibraryofHamiltoniansfor benchmarking quantum algorithms and hardware.Quan- t...
work page 2024
- [38]
-
[39]
Alvertis, Alan Bidart, Ben DalFavero, Sophia E
Ryan LaRose, Antonios M. Alvertis, Alan Bidart, Ben DalFavero, Sophia E. Economou, J. Wayne Mullinax, Mafalda Ramôa, Jeremiah Rowland, Brenda Rubenstein, Nicolas PD Sawaya, Prateek Vaish, Grant M. Rotskoff, and Norm M. Tubman. The cost of quantum algorithms for biochemistry: A case study in metaphosphate hydrol- ysis, February 2026
work page 2026
- [40]
-
[41]
Practical Quantum Error Mitigation for Near-Future Applications
SuguruEndo, SimonC.Benjamin, andYingLi. Practical Quantum Error Mitigation for Near-Future Applications. Physical Review X, 8(3):031027, July 2018
work page 2018
- [42]
-
[43]
Overlapped grouping GitHub.https://github.com/jdrowland/ OverlappedGroupingNumerics/
Jeremiah Rowland. Overlapped grouping GitHub.https://github.com/jdrowland/ OverlappedGroupingNumerics/. Appendix A: F urther discussion of zero covariance assumption Next we show that for Hamiltonians with a large spec- tral width, there exist states for which covariance domi- nates. Large spectral width is often associated with long- range order or stron...
-
[44]
Moreover, lettingE min andE max denote the minimum and maximum eigenvalues ofHand W :=E max −E min, there exists a state|ψ⋆⟩such that Var(H) ψ⋆ = W 2 4 and D(ψ⋆) Var(H)|ψ⋆ ≤ 4∥c∥2 2 W 2 . Proof.Since eachP i has±1outcomes,Var(P i) = 1− ⟨Pi⟩2 ≤1, hence D(ψ) = X i c2 i Var(Pi)≤ X i c2 i =∥c∥ 2 2. Let|E min⟩and|E max⟩be eigenstates achievingE min and Emax, a...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.