REVIEW 3 major objections 5 minor 47 references
From Circuits to Hardware: Benchmarking Standard and Qubit-Efficient Quantum Optimization on Real Hardware
T0 review · 3 major / 5 minor · reviewed 2026-07-14 · grok-4.5
Pith's one-line read On present hardware, gate-based quantum optimizers become noise-dominated near 770 two-qubit gates, while dense assignment problems yield no feasible solutions at all.
desk verdict Solid, first-of-its-kind multi-family hardware baseline for quantum optimization; the F_est thresholds and QAP/QAOA claims hold up under the stated protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The independent-error gate-count fidelity proxy F_est ≈ (1 − ε_2Q)^N_2Q, built from transpiled two-qubit gate counts and backend calibration data, used as a diagnostic to separate signal-preserving from noise-dominated hardware regimes.
What would settle it
Re-run the same MDKP and MIS circuits after stronger routing or on lower-error hardware so F_est exceeds 0.1, and test whether QAOA-family solutions then systematically beat the matched random control and whether any method returns a feasible QAP assignment.
Extended reading notes
Core claim
Across 247 method–instance combinations on Heron-family processors, an independent-error gate-count fidelity near F_est ≈ 0.1 marks the onset of noise-dominated execution for multi-dimensional knapsack and maximum independent set; no tested hardware method produces a feasible quadratic-assignment solution; and most feasible low-fidelity QAOA-family outcomes lie within a matched uniform-random best-shot range under the same selection and local-refinement rules.
Load-bearing premise
A simple product of two-qubit error rates over gate count, under a fixed short optimizer budget, is enough to mark when hardware results stop carrying real algorithmic signal.
Editorial extensions
If this is right
- Deployability should be judged by transpiled two-qubit count times gate error, not by qubit count alone.
- Dense one-hot assignment problems remain outside the reliable operating regime of the tested end-to-end pipelines.
- Deeply compiled QAOA-family implementations under this protocol should not be assumed to beat best-of-budget random selection.
- Future hardware studies must report feasibility, transpiled resources, and a fidelity proxy alongside objective values.
- Qubit-efficient encodings help only when the compressed circuits stay inside the empirical fidelity budget after routing and decoding.
Reading between the lines
- As two-qubit error rates fall, the ~770-gate threshold will move outward, but only if compilation expansion also shrinks for structured ansätze.
- Encoding changes that enlarge the feasible-manifold fraction, not just width reduction, are likely required before QAP-like problems become hardware-accessible.
- For the instance sizes tested, classical solvers remain the practical solvers; the lasting contribution is the diagnostic protocol itself.
- Negative simulator–hardware gaps on some packing instances suggest finite-shot selection and trajectory effects can dominate idealized noise models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a unified real-hardware benchmark of gate-based quantum optimization methods (VQE, CVaR-VQE, standard/multi-angle/warm-start QAOA, PCE, and QRAO) across four NP-hard combinatorial problems (MDKP, MIS, QAP, MSP) on IBM Heron r1/r2 processors under a common fixed-budget protocol with resilience-level-2 mitigation. Across 247 method–instance combinations the authors report transpiled circuit resources, an independent-error gate-count fidelity proxy F_est, feasibility, and solution quality relative to classical references, together with matched uniform-random controls, MPS stability audits, and compilation-only counterfactuals. The central empirical claims are that an operating point near F_est ≈ 0.1 marks the onset of noise-dominated execution for MDKP and MIS, that no tested method recovers a feasible QAP assignment (attributed to dense one-hot encodings and an exponentially sparse feasible manifold), that most low-fidelity QAOA-family outcomes are compatible with random best-of-budget sampling, and that qubit-efficient encodings extend runnable sizes only within the empirical fidelity budget.
Significance. If the reported regimes hold, the work supplies a timely, reproducible empirical baseline that the community currently lacks: multi-family, multi-problem hardware results under one protocol, including the first real-hardware QRAO data and the first multi-problem PCE hardware benchmark. Strengths include the explicit F_est diagnostic tied to calibration data, the matched random-control design (Appendix N), the compilation counterfactuals that leave deep QAOA circuits below F_est = 10^{-3}, the MPS bond-cap audits, full backend-aware resource tables, and public code. These elements make the paper more than a collection of runs; they give practitioners concrete decision guidance (Table 20) and a diagnostic framework that can track progress as hardware improves. The honesty about scope (tested implementations, not QAOA in general; fixed-budget protocol rather than best-case) further increases its value as a reference.
major comments (3)
- Section 4.5 and Appendix C: the common protocol is a single COBYLA trajectory with a hard 200-evaluation cap and uniform [0, 2π] initialization. The reduced multi-seed simulator study (Table 26) already shows material initialization sensitivity for VQE and WS-QAOA (gaps ranging 8–50 % on MIS 1tc.32). Because the headline tables report only one primary execution per method–instance pair, the cross-family ranking is protocol-conditioned rather than robust. A short additional multi-seed hardware or high-fidelity simulator panel on a few representative instances (or an explicit statement that all claims are strictly under the stated fixed-budget protocol) is needed to keep the comparative claims load-bearing.
- Section 4.8 and the hardware-diagnostics appendices: backend heterogeneity (ibm_fez, ibm_torino, ibm_marrakesh) is acknowledged but not quantified. Different topologies, calibration windows, and routing overheads can shift both N_2Q and realized F_est. For the key claims that rest on absolute gate counts (the F_est ≈ 0.1 operating point and the QAOA random-control comparison), a brief sensitivity check—e.g., re-transpiling a subset of circuits under a single fixed backend calibration snapshot—would strengthen the claim that the observed regimes are not artifacts of device assignment.
- Section 5.4 and Appendix A: the QAP infeasibility conclusion is strong and well-supported by the feasible-fraction calculation (10!/2^100 ≈ 10^{-23.54}). However, only the direct one-hot encoding is tested. Because the paper’s broader thesis is that encoding choice interacts with hardware constraints, a short discussion (or a single alternative encoding experiment, even if only in simulation) of whether a more compact or hierarchical assignment encoding could move any QAP instance into the F_est ≳ 0.1 regime would make the structural-barrier claim more complete.
minor comments (5)
- Figure 5 caption and Section 5.8: the upper x-axis uses a fixed ε_2Q = 0.003 reference; it would help readers if the caption explicitly stated that backend-specific F_est values (Appendix O) can differ slightly from this visual reference.
- Table 1 and Appendix A: the coefficient dynamic-range column mixes scientific notation inconsistently (e.g., 8.53×10^3 vs 2.21×10^10); a uniform format would improve readability.
- Section 3.5: the MSP quality metrics (TDev, MDev, AR_TDev) are well-motivated, but a one-sentence reminder that AR_TDev = 1 is the certified optimum would help readers who jump directly to the tables.
- Appendix D: the multi-reoptimization and dynamic-perturbation enhancements are described in detail, yet the main-text PCE results use only the single-pass protocol. A clearer cross-reference early in Section 4.6 would prevent readers from assuming the full enhancement suite was applied.
- Minor typographical inconsistencies appear in a few places (e.g., “ans¨ atze”, “to ising()”, occasional missing spaces around em-dashes). A final copy-edit pass would polish the manuscript.
Circularity Check
No significant circularity: empirical hardware benchmark with independent diagnostics, analytical penalties, and matched random controls.
full rationale
The paper is a fixed-protocol empirical comparison of VQE/CVaR-VQE, QAOA-family, PCE, and QRAO across MDKP, MIS, QAP, and MSP on IBM Heron hardware. Load-bearing claims rest on (i) feasibility and gap-to-BKS against classical references, (ii) transpiled N_2Q and backend calibration, (iii) the independent-error proxy F_est ≈ (1−ε_2Q)^N_2Q used only as a diagnostic, not as a fitted predictor of rankings, and (iv) a matched uniform-random best-shot control for low-fidelity QAOA-family runs. Penalty coefficients are fixed analytically by the pre-penalty objective-bound rule before any quantum execution and are not tuned to hardware outcomes. Thresholds F_est≈0.1 and 0.01 are explicitly labeled benchmark-specific empirical reference points, not first-principles predictions. Compilation counterfactuals hold circuits and parameters fixed and only re-route. No self-definitional loop, fitted-input-as-prediction, load-bearing self-citation uniqueness claim, or renaming of a known result as a derivation was found. The work is self-contained against external classical BKS and random baselines.
Assumptions & free parameters
free parameters (5)
- CVaR confidence level α =
0.25
- QRAO compression parameter k =
3
- Optimizer evaluation budget =
200 evals / 1000 shots
- F_est diagnostic thresholds =
0.1 / 0.01
- Automatic QUBO penalty multiplier λ =
instance-specific (Eq. 6)
assumptions (4)
- domain assumption Independent, identically distributed depolarizing errors on two-qubit gates dominate the process-fidelity estimate F_est ≈ (1 − ε_2Q)^N_2Q.
- domain assumption The Qiskit automatic penalty construction (Eq. 6) produces a valid unconstrained QUBO whose low-energy states correspond to high-quality feasible solutions of the original constrained problem.
- ad hoc to paper A single fixed-budget COBYLA trajectory with uniform [0, 2π] initialization is a fair common protocol for cross-method comparison.
- standard math Classical best-known or certified optima (BKS) from the cited libraries are correct reference values for gap reporting.
invented entities (1)
-
Independent-error gate-count fidelity proxy F_est
independent evidence
Cite this review
Pith. "Pith review of From Circuits to Hardware: Benchmarking Standard and Qubit-Efficient Quantum Optimization on Real Hardware." pith.science (2026). https://pith.science/paper/SVNXUMZ3
@misc{pith2026260711637,
author = {Pith},
title = {Pith review of: From Circuits to Hardware: Benchmarking Standard and Qubit-Efficient Quantum Optimization on Real Hardware},
year = {2026},
howpublished = {\url{https://pith.science/paper/SVNXUMZ3}},
note = {Machine review of arXiv:2607.11637}
}
abstract
Despite rapid progress in quantum optimization, broad real-hardware benchmarks comparing multiple algorithmic families across diverse combinatorial problems under a common protocol remain limited. We benchmark gate-based quantum optimization on four NP-hard problems: multi-dimensional knapsack (MDKP), maximum independent set (MIS), quadratic assignment (QAP), and market-share (MSP). We study VQE, CVaR-VQE, standard, multi-angle, and warm-start QAOA, together with qubit-efficient PCE and QRAO, on IBM Heron r1/r2 processors using resilience-level-2 mitigation. To our knowledge, this includes the first real-hardware QRAO results and the first multi-problem PCE hardware benchmark. Across 247 method-instance combinations, we report transpiled circuit size, hardware outcomes, and an independent-error gate-count fidelity proxy, $F_{\mathrm{est}}$. For MDKP and MIS, an empirical operating point near $F_{\mathrm{est}}\approx 0.1$, corresponding to about 770 two-qubit gates at the median Heron-r2 CZ error rate, marks the onset of noise-dominated execution. QAP exposes a separate bottleneck: dense one-hot encodings and an exponentially sparse feasible manifold, with feasible fraction $10!/2^{100}$ at $n=10$; no tested hardware method produces a feasible assignment. Compiled QAOA-family circuits are generally noise dominated, and a matched uniform-random control shows that most feasible low-fidelity outcomes fall within the random range, apart from one finite-sample MIS warm-start exception. A SWAP-aware, fractional-gate, Nighthawk-topology compilation counterfactual reduces two-qubit counts but leaves all circuits below $F_{\mathrm{est}}=10^{-3}$. These conclusions apply to the tested implementations rather than QAOA in general. Qubit-efficient methods extend runnable instance sizes, but only within the empirical fidelity budget.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Abbas A, Ambainis A, Augustino B, B¨ artschi A, Buhrman H, Coffrin C J, Cortiana G, Dunjko V, Egger D J, Elmegreen B Get al.2023 Quantum optimization: Potential, challenges, and the path forward Tech. rep. Los Alamos National Laboratory (LANL)
2023
-
[2]
Peruzzo A, McClean J, Shadbolt P, Yung M H, Zhou X Q, Love P J, Aspuru-Guzik A and O’Brien J L 2014Nature Communications5ISSN 2041-1723 URL http://dx.doi.org/10.1038/ncomms5213
-
[3]
Barkoutsos P K, Nannicini G, Robert A, Tavernelli I and Woerner S 2020Quantum4256 ISSN 2521-327X URLhttp://dx.doi.org/10.22331/q-2020-04-20-256
-
[4]
Farhi E, Goldstone J and Gutmann S 2014arXiv preprint arXiv:1411.4028
-
[5]
Sciorilli M, Borges L, Patti T L, Garc´ ıa-Mart´ ın D, Camilo G, Anandkumar A and Aolita L 2025Nature Communications16ISSN 2041-1723 URL http://dx.doi.org/10.1038/s41467-024-55346-z
-
[6]
Fuller B, Hadfield C, Glick J R, Imamichi T, Itoko T, Thompson R J, Jiao Y, Kagele M M, Blom-Schieber A W, Raymond Ret al.2024IEEE Transactions on Quantum Engineering5 1–15
-
[7]
Koch T, Bernal Neira D E, Chen Y, Cortiana G, Egger D J, Heese R, Hegade N N, Gomez Cadavid A, Huang R, Itoko Tet al.2025arXiv e-printsarXiv–2504
-
[8]
Lubinski T, Johri S, Varosy P, Coleman J, Zhao L, Necaise J, Baldwin C H, Mayer K and Proctor T 2023IEEE Transactions on Quantum Engineering41–32 ISSN 2689-1808 URL http://dx.doi.org/10.1109/TQE.2023.3253761
Show all 47 references
-
[9]
Dupont M, Sundar B, Evert B, Neira D E B, Peng Z, Jeffrey S and Hodson M J 2025Physical Review Applied23ISSN 2331-7019 URL http://dx.doi.org/10.1103/PhysRevApplied.23.014045
-
[10]
Brandhofer S, Braun D, Dehn V, Hellstern G, H¨ uls M, Ji Y, Polian I, Bhatia A S and Wellens T 2022Quantum Information Processing22ISSN 1573-1332 URL http://dx.doi.org/10.1007/s11128-022-03766-5
-
[11]
Li A C Y, Alam M S, Iadecola T, Jahin A, Job J, Kurkcuoglu D M, Li R, Orth P P, ¨Ozg¨ uler A B, Perdue G N and Tubman N M 2023Physical Review Research5ISSN 2643-1564 URL http://dx.doi.org/10.1103/PhysRevResearch.5.033071
-
[12]
Shaydulin R, Li C, Chakrabarti S, DeCross M, Herman D, Kumar N, Larson J, Lykov D, Minssen P, Sun Y, Alexeev Y, Dreiling J M, Gaebler J P, Gatterman T M, Gerber J A, Gilmore K, Gresh D, Hewitt N, Horst C V, Hu S, Johansen J, Matheny M, Mengle T, Mills M, Moses S A, Neyenhuis B...
-
[14]
Chekuri C and Khanna S 2004SIAM Journal on Computing33837–851
-
[15]
Lawler E L, Lenstra J K and Rinnooy Kan A 1980SIAM Journal on Computing9558–565
-
[16]
Sahni S and Gonzalez T 1976Journal of the ACM23555–565
-
[17]
Kellerer H, Pferschy U and Pisinger D 2004Multidimensional Knapsack Problems(Berlin, Heidelberg: Springer Berlin Heidelberg) pp 235–283 ISBN 978-3-540-24777-7 URL https://doi.org/10.1007/978-3-540-24777-7_9
-
[18]
Garey M R and Johnson D S 1979Computers and Intractability: A Guide to the Theory of NP-Completeness(W. H. Freeman and Company)
-
[19]
Koopmans T C and Beckmann M 1957Econometrica2553–76 ISSN 00129682, 14680262 URLhttp://www.jstor.org/stable/1907742 82 IOP PublishingJournalvv(yyyy) aaaaaa Sharma & Lau
-
[20]
Cornu´ ejols G and Dawande M 1999INFORMS Journal on Computing11205–210 URL https://doi.org/10.1287/ijoc.11.2.205
-
[21]
Cerezo M, Arrasmith A, Babbush R, Benjamin S C, Endo S, Fujii K, McClean J R, Mitarai K, Yuan X, Cincio L and Coles P J 2021Nature Reviews Physics3625–644 ISSN 2522-5820 URL http://dx.doi.org/10.1038/s42254-021-00348-9
-
[22]
Blekos K, Brand D, Ceschini A, Chou C H, Li R H, Pandya K and Summer A 2024Physics Reports10681–66 ISSN 0370-1573 URL http://dx.doi.org/10.1016/j.physrep.2024.03.002
2024 doi
-
[23]
Tilly J, Chen H, Cao S, Picozzi D, Setia K, Li Y, Grant E, Wossnig L, Rungger I, Booth G H and Tennyson J 2022Physics Reports9861–128 ISSN 0370-1573 URL http://dx.doi.org/10.1016/j.physrep.2022.08.003
2022 doi
-
[24]
Cerezo M, Sone A, Volkoff T, Cincio L and Coles P J 2021Nature Communications12ISSN 2041-1723 URLhttp://dx.doi.org/10.1038/s41467-021-21728-w
-
[25]
Egger D J, Mareˇ cek J and Woerner S 2021Quantum5479 ISSN 2521-327X URL http://dx.doi.org/10.22331/q-2021-06-17-479
2021 doi
-
[26]
Herrman R, Lotshaw P C, Ostrowski J, Humble T S and Siopsis G 2022Scientific Reports12 6781
-
[27]
McDowall K, Kapourniotis T, Oliver C, Lolur P and Georgopoulos K 2025arXiv preprint arXiv:2504.06885
-
[28]
Turati G, Dacrema M F and Cremonesi P 2023 Benchmarking adaptative variational quantum algorithms on qubo instances (Preprint2308.01789) URL https://arxiv.org/abs/2308.01789
2023
-
[29]
Schw¨ agerl T, Chai Y, Hartung T, Jansen K and K¨ uhn S 2026Quantum Machine Intelligence 8ISSN 2524-4914 URLhttp://dx.doi.org/10.1007/s42484-026-00355-y
-
[30]
Fr´ eville A 2004European Journal of Operational Research1551–21
-
[31]
Butenko S and Pardalos P M 2002Handbook of Applied Optimization403–413
-
[32]
Loiola E M, de Abreu N M M, Boaventura-Netto P O, Hahn P and Querido T 2007European Journal of Operational Research176657–690
-
[33]
Burkard R E, Karisch S E and Rendl F 1997 Qaplib – a quadratic assignment problem library https://coral.ise.lehigh.edu/data-sets/qaplib/accessed: 2025-02-12
1997
-
[34]
Cornuejols G and Dawande M 1999Manufacturing and Service Operations ManagementAlso described in QOBLIB benchmark suite
-
[35]
Puchinger J, Raidl G R and Pferschy U 2010INFORMS Journal on Computing22250–265
-
[36]
Drake J 2015 Benchmark instances for the multidimensional knapsack problem
2015
-
[37]
Sloane N J A 2000 Challenge problems: Independent sets in graphs accessed: January 14, 2025 URLhttps://oeis.org/A265032/a265032.html
2000
-
[38]
Montanez-Barrera J A, Ji Y, von Spakovsky M R, Neira D E B and Michielsen K 2025 Optimizing qaoa circuit transpilation with parity twine and swap network encodings (Preprint 2505.17944) URLhttps://arxiv.org/abs/2505.17944
2025 arXiv
-
[39]
Monta˜ nez-Barrera J A and Michielsen K 2025npj Quantum Information11ISSN 2056-6387 URLhttp://dx.doi.org/10.1038/s41534-025-01082-1
-
[40]
McDowall K, Georgopoulos K and Wallden P 2026 Spectral gap informed ramp qaoa (Preprint 2604.24580) URLhttps://arxiv.org/abs/2604.24580
2026 arXiv
-
[41]
Lucas A 2014Frontiers in Physics25
-
[42]
Kochenberger G, Hao J K, Glover F, Lewis M, L¨ u Z, Wang H and Wang Y 2014Journal of Combinatorial Optimization2858–81 83 IOP PublishingJournalvv(yyyy) aaaaaa Sharma & Lau
-
[43]
Javadi-Abhari A, Treinish M, Krsulich K, Wood C J, Lishman J, Gacon J, Martiel S, Nation P D, Bishop L S, Cross A Wet al.2024arXiv preprint arXiv:2405.08810
-
[44]
Available athttps://www.ibm.com/quantum/hardwareaccessed during experimental window of this study
IBM Quantum 2024 Heron r2: A 156-qubit quantum processor with TLS mitigation and enhanced coherence IBM Quantum Developer Conference announcement, November 2024. Available athttps://www.ibm.com/quantum/hardwareaccessed during experimental window of this study
2024
-
[45]
IBM Quantum 2025 IBM Quantum Developer Roadmap: Nighthawk, Heron, and beyond Available athttps://www.ibm.com/quantum/technologyNighthawk processor targets approximately 5000 two-qubit gates initially, with subsequent iterations projected for 7500 (2026), 10000 (2027), and 1500...
2025
-
[46]
Temme K, Bravyi S and Gambetta J M 2017Physical Review Letters119180509
-
[47]
Endo S, Benjamin S C and Li Y 2018Physical Review X8ISSN 2160-3308 URL http://dx.doi.org/10.1103/PhysRevX.8.031027
-
[48]
Polj´ ak S and Turz´ ık D 1986Journal of Combinatorial Theory, Series B4099–103 84
Reviewed July 14, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.