Pith. sign in

REVIEW 3 major objections 4 minor 4 cited by

A Review and Collection of Metrics and Benchmarks for Quantum Computers: definitions, methodologies and software

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This review attempts to give quantum computing a shared benchmarking language, curating metrics into ten categories and defining each in a uniform, software-linked format.

desk verdict Useful compendium with a real definitional error in the diamond-norm entry; worth reviewing and fixing before it becomes a standard reference. read the letter →

arxiv 2502.06717 v1 pith:EIFLRHCA submitted 2025-02-10 quant-ph

Deep Lall , Abhishek Agarwal , Weixi Zhang , Lachlan Lindoy , Tobias Lindström , Stephanie Webster , Simon Hall , Nicholas Chancellor
show 10 more authors
Petros Wallden Raul Garcia-Patron Elham Kashefi Viv Kendon Jonathan Pritchard Alessandro Rossi Animesh Datta Theodoros Kapourniotis Konstantinos Georgopoulos Ivan Rungger
This is my paper · ORCID
classification quant-ph
keywords quantumcomputingbenchmarksperformancemetricsrandomizedbenchmarkingvolumegatesettomographyreproducibilitystandardizationannealers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This review attempts to give quantum computing a shared benchmarking language. It curates a set of metrics running from qubit relaxation times to application-level tasks, grouped into ten categories, and presents each metric in the same template: definition, step-by-step methodology, explicit assumptions and limitations, and a link to open-source software that evaluates it. The aim is that when vendors and researchers report metrics in this consistent format, devices can be compared fairly and holistically rather than through cherry-picked numbers. The authors also identify five areas where international standardization working groups could begin.

What carries the argument

The organizing device is a metric template repeated across ten categories M1 through M10: each entry contains a definition, a step-by-step measurement methodology, assumptions and limitations, references, and a pointer to software. Two cross-cutting technical backbones are the volumetric benchmarking framework (test circuits of varying width and depth with defined success criteria), which underlies the circuit-execution metrics, and gate set tomography, which supplies the process fidelities and SPAM metrics for small qubit counts. The linked open-source software repository ties every metric to a concrete implementation.

What would settle it

Attempt to reproduce a metric from the collection using the linked software: for example, run the randomized benchmarking metric for a single qubit under the paper's stated emulator noise model. If the repository is inaccessible, lacks the described implementations, or produces values that conflict with the document's methodology, the paper's reproducibility claim is not met.

Watch

Extended reading notes

Core claim

The central claim is that holistic and reproducible benchmarking of quantum computers can be reached today by pairing a finite metric collection with a uniform definition format and linked open-source implementations. The collection spans hardware architecture properties, qubit and gate quality, circuit and task execution quality, speed, stability, plus metrics for quantum annealers, boson samplers, and neutral-atom analogue simulators. For well-established metrics the authors argue the community should move to standardized methodology, inter-laboratory comparisons, and shared reporting of data and software, while leaving still-evolving metrics open for development.

Load-bearing premise

The reproducibility of every metric depends on the linked online software repository existing, being complete, and correctly implementing the methodology described for each metric; the preprint itself ships no code, so if the repository is missing, incomplete, or buggy, the reproducibility claim fails while the metric definitions may still stand.

Editorial extensions

If this is right

  • Hardware vendors could publish a standardized metric card covering all categories, giving cloud users a fair basis for service choice.
  • Researchers could evaluate every metric under a single consistent noise model and study correlations between them.
  • Standardization bodies could use the five proposed work areas as a concrete starting agenda.
  • Non-gate-based platforms such as annealers, boson samplers, and neutral-atom simulators gain a first candidate set of metrics that can be refined as the technology matures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reporting guidelines gain traction, requiring disclosure of executed circuits and classically optimized circuits could become a de facto transparency norm for vendors.
  • The same metric template could later be applied to logical qubits as quantum error correction matures, giving the collection a natural upgrade path.
  • The proposed inter-laboratory comparisons for T1, T2, and randomized benchmarking could evolve into a certification-like procedure analogous to classical benchmarking standards.
  • Readers could use the paper's documented emulator noise model to construct a synthetic device profile and compare vendor-reported metric values against it, providing a quick independent sanity check.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript is a review and compendium of metrics and benchmarks for quantum computers. It proposes a structured collection, organized as categories M1--M10, covering hardware architecture properties, qubit/gate/circuit/task quality, speed, stability, and metrics for quantum annealers, boson-sampling devices, and neutral-atom analogue devices. For each metric the paper gives a common-format definition, a step-by-step methodology, assumptions and limitations, references, and a link to open-source software (QCMet, qcmet.npl.co.uk). It also provides background on quantum computing models and hardware platforms, and proposes five areas for international standardization. The central claims are: (1) the collection is comprehensive and enables holistic comparisons; (2) the definitions are consistently formatted with transparent methodology and limitations; and (3) the approach is reproducible through linked open-source software.

Significance. If the claims were fully supported, this would be a valuable common reference for hardware users, manufacturers, and standardization bodies. The paper's main strength is organizational: the metric descriptions in Part IV follow a uniform template, and the sections I could check faithfully restate standard methodologies from the cited literature. The paper is also careful to list key assumptions and limitations, such as Markovian noise for randomized benchmarking and the classical-simulation bottleneck for quantum volume. The plan to link every metric to a single open-source implementation is a concrete step toward reproducibility, and the paper explicitly names the software libraries it builds on. However, the article contains no new derivations or experimental data, so its value depends entirely on the accuracy of the metric definitions and on the availability and correctness of the external software. A key definition error, together with unsupported claims of comprehensiveness and reproducibility, currently prevent the manuscript from serving as a reliable reference.

major comments (3)
  1. [II.2, M3.2; also Part IV M3.2] The description of the diamond norm is internally contradictory and incorrect. The text says the diamond norm is 'the minimum trace distance, over all possible initial states, of the states resulting from the application of the actual and ideal process', immediately adding that it 'measures the maximum distinguishability'. The standard definition (Watrous; Nielsen & Chuang; the references cited in this entry) is the maximum trace distance between the outputs of the two channels, maximized over all input states, including states with an ancillary system. The 'minimum' version is not the diamond norm and is not a meaningful worst-case figure of merit. If any implementation in the QCMet software follows the printed definition literally, all reported diamond-norm values would be wrong. This must be corrected in both the short description and the detailed metric entry, with the maximization over entangled input states stated explicitly.
  2. [II.1 and II.3 (Discussion and outlook)] The central claim of a 'comprehensive collection allowing holistic comparisons' is not supported by an explicit selection protocol. The paper states that the collection is 'by design restricted to a limited number' of metrics, and the discussion names several candidate metrics that are not included, such as cross-entropy benchmarking, CLOPS, and QEC-specific metrics. However, no criterion is given for why the chosen set is sufficient for a holistic evaluation while the excluded ones are not. As written, the comprehensiveness claim is an assertion rather than a demonstrated property. I recommend either adding a section that justifies the inclusion/exclusion criteria and explains what 'holistic' means operationally, or softening the claim to 'a representative collection'.
  3. [I (Introduction) and III, Sec. 6.1 (noise model); QCMet link] The reproducibility claim depends entirely on the external repository qcmet.npl.co.uk. The preprint ships no code, no data, no commit hash, and no version identifier, and I could not verify from the manuscript that the repository exists or that it implements every metric methodology exactly as described. Since 'reproducible approach' is one of the three central claims, the manuscript should provide a permanent identifier (DOI or release number), the repository license, and a minimal smoke-test account showing that each metric entry has a corresponding executable implementation consistent with the printed methodology. Without this, the reader cannot distinguish a documented methodology from a claimed-but-unverified software artifact.
minor comments (4)
  1. [Throughout] There are repeated typographical and grammatical errors, including 'Deustch' for Deutsch, 'comprized' for comprised, 'the the' duplication, 'time take' for 'time taken', and a dangling list item in Sec. 6.1 following the noise model ('for all of the 2-qubi CX gates'). A careful proofreading pass is needed.
  2. [Fig. 6.1.1 and Sec. 6.1] The text says the figure shows 'X and Ry(π/4) gates', while the caption states 'Ry(−π/2)'. One of these is wrong; please align the text, caption, and the actual Bloch-sphere illustration.
  3. [Eq. (6.1.13)] The sentence describing the depolarizing channel says the qubit does nothing with probability '1−p', but the equation and surrounding text use γD. This should be '1−γD'.
  4. [II.2, M2.3] The entry for the idle qubit purity oscillation frequency is described only in prose; no operational definition is given for how the frequency is extracted from data. Since this metric is claimed to indicate non-Markovian noise, a precise definition or a pointer to the Part IV entry with the actual estimator would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the review organizes externally established metrics and benchmarks; its central claims do not reduce to fitted inputs, self-citation chains, or definitional loops.

full rationale

This paper is a review and compendium rather than a predictive derivation. Its central deliverables are: (1) a categorized collection of metrics, (2) a consistent definitional format with stated assumptions and limitations, and (3) links to open-source software for reproducibility. None of these deliverables is derived from data fitted to its own claims. Each metric definition is traced to established external sources, e.g. the diamond norm to Refs. [74,80], randomized benchmarking to Refs. [81,82,86,91,92], quantum volume to Refs. [109-111], and gate set tomography to Refs. [101-104]. The methodology sections describe standard practice, and the paper explicitly acknowledges limitations, such as the potential bias in volumetric benchmarking success criteria and the Markovian-noise assumption in GST. No equation is shown to be definitionally equal to its own input, and no fitted parameter is renamed as a prediction. The only self-referential element is the QCMet software repository (qcmet.npl.co.uk) and the statement that "The software directly implements the methodology described in this document." This is a reproducibility-supporting claim, not a load-bearing logical premise for a derived result. If the repository were missing or incorrect, the paper's practical reproducibility promise would be weakened, but the metric definitions would still stand on their cited external sources; this is a tooling risk, not circularity. No uniqueness theorem from the authors is invoked to forbid alternatives, and no central argument reduces to a self-citation chain. One internal inconsistency should be flagged for correctness rather than circularity: metric M3.2 describes the diamond norm as "the minimum trace distance, over all possible initial states," while the standard definition is the maximum trace distance; the same paragraph correctly says the metric "measures the maximum distinguishability." This contradiction undermines the paper's consistency claim and should be corrected before the collection is used as a normative reference. It is not, however, a circular step: the definition does not depend on the paper's own conclusions, and it does not fit a parameter to a later prediction. The circularity score therefore remains 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities or derived constants. Its numerical inputs are hand-chosen emulator noise parameters that illustrate the methodology but do not support the standardization claims.

free parameters (5)
  • Emulator T1 mean = 50 µs
    Hand-chosen to mimic superconducting qubits in the noise model (Sec. 6.1); illustrative only, not fitted to data.
  • Emulator T2 mean = 70 µs
    Hand-chosen; illustrative only.
  • Single-qubit depolarizing rate = 0.0005
    Chosen for Rx(pi/2) gates in the emulator noise model; illustrative only.
  • Two-qubit depolarizing rate = 0.005
    Chosen for CX gates; illustrative only.
  • Coherent error angles = pi/100, pi/120, pi/100
    Over-rotation and phase errors set by hand in the emulator; illustrative only.
assumptions (4)
  • standard math Quantum operations can be described as completely positive trace-preserving maps (Kraus representation).
    Used throughout the metric definitions (Sec. 6.1); standard formalism.
  • domain assumption The cited literature correctly defines the metrics as summarized in this collection.
    The paper restates metric definitions from references; if any restatement is inaccurate, the collection's reliability is affected.
  • domain assumption Open-source libraries Qiskit and PyGSTi correctly implement the referenced protocols.
    The software implementation is based on these libraries; errors in the libraries would affect the reproducibility claim.
  • ad hoc to paper The selected metrics are sufficient to holistically benchmark device performance across platforms.
    The choice of M1-M10 is an editorial decision; there is no objective proof that it covers all relevant performance aspects or avoids platform bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Review and Collection of Metrics and Benchmarks for Quantum Computers: definitions, methodologies and software." pith.science (2026). https://pith.science/paper/EIFLRHCA

@misc{pith2026250206717,
  author       = {Pith},
  title        = {Pith review of: A Review and Collection of Metrics and Benchmarks for Quantum Computers: definitions, methodologies and software},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EIFLRHCA}},
  note         = {Machine review of arXiv:2502.06717}
}
read the original abstract

Quantum computers have the potential to provide an advantage over classical computers in a number of areas. Numerous metrics to benchmark the performance of quantum computers, ranging from their individual hardware components to entire applications, have been proposed over the years. Navigating the resulting extensive literature can be overwhelming. Objective comparisons are further hampered in practice as different variations of the same metric are used, and the data disclosed together with a reported metric value is often not sufficient to reproduce the measurements. This article addresses these challenges by providing a review of metrics and benchmarks for quantum computers and 1) a comprehensive collection of benchmarks allowing holistic comparisons of quantum computers, 2) a consistent format of the definitions across all metrics including a transparent description of the methodology and of the main assumptions and limitations, and 3) a reproducible approach by linking the metrics to open-source software used to evaluate them. We identify five areas where international standardization working groups could be established, namely: i) the identification and agreement on the categories of metrics that comprehensively benchmark device performance; ii) the identification and agreement on a set of well-established metrics that together comprehensively benchmark performance; iii) the identification of metrics specific to hardware platforms, including non-gate-based quantum computers; iv) inter-laboratory comparison studies to develop best practice guides for measurement methodology; and v) agreement on what data and software should be reported together with a metric value to ensure trust, transparency and reproducibility. We provide potential routes to advancing these areas. We expect this compendium to accelerate the progress of quantum computing hardware towards quantum advantage.

Figures

Figures reproduced from arXiv: 2502.06717 by the authors.

Figure 1.0
Figure 1.0. 1.: Schematic of the collection of metrics presented in this document, with each category illustrated in a different color. entangled indirectly by swapping the states of qubits, then applying the direct entangling operation between a pair of directly connected qubits, and then swapping the states of qubits back. The swapping operations can be done either by physically moving around the qubits [9, 19, 23, 115] or by… view at source ↗
Figure 4.2
Figure 4.2. 1.: Example circuit diagram for a three-qubit quantum circuit. Each horizontal wire represents a qubit, and the initial state is shown at the left end of the circuit. Symbols and boxes represent X, Ry, Hadamard and CX gates, with measurements at the end of the circuit. The mathematical definitions for these gates and their operation are described in section 6.1. Multi-qubit gates are drawn vertically across wires, w… view at source ↗
Figure 6.1
Figure 6.1. 1.: Bloch sphere visualizations of (a) a general qubit state |ψ⟩ = cos (θ)|0⟩ + e iϕ sin (θ)|1⟩ as green arrow, (b) an X gate applied to this |ψ⟩, leading to |ψ1⟩ = X |ψ⟩, and (c) an Ry(−π/2) gate applied to |ψ1⟩, leading to |ψ2⟩ = Ry(−π/2)|ψ1⟩. Quantum gate operations on qubits are described by unitary operators. Denoting a unitary operator by U, it evolves a state vector ψ into U |ψ⟩. Unitary operators preserve th… view at source ↗
Figures from the paper (2 more)
Figure 6.3
Figure 6.3. Figure 6.3: 1.: Circuit diagrams showing the circuit structure of all long-sequence gate set tomography (LSGST) circuits for an arbitrary number of qubits. For both circuits, each fiducial circuit is colored orange, and the germ circuit is colored green. (a) First the fiducial c…
Figure 6.3
Figure 6.3. Figure 6.3: 2.: Example PTMs of (a) an ideal CX gate, and (b) a noisy CX gate, obtained by GST. The noisy CX has many small non-zero elements in the PTM that arise due to noise. The color represents the value of the PTM element. For increased clarity, the size of the squares is …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fast-tracking and disentangling of qubit noise fluctuations using minimal-data averaging and hierarchical discrete fluctuation auto-segmentation

    quant-ph 2025-05 conditional novelty 8.0 of 10

    A new hierarchical segmentation algorithm disentangles concurrent discrete frequency fluctuations in transmons, tracking them at tens of milliseconds resolution and attributing them to charge parity switching and a ch...

  2. Practical Insights into Fair Comparison and Evaluation Frame for Neutral-Atom Compilers

    cs.ET 2026-04 unverdicted novelty 7.0 of 10

    Under a unified evaluation framework, the 415.8x DasAtom-over-Enola QFT30 gap reported in prior work becomes 8.1x (radius 6) or 4.86x (radius 2), and 3.26x after collapsing RSQASM-redundant Enola movements.

  3. Clifford Volume and Free Fermion Volume: Complementary Scalable Benchmarks for Quantum Computers

    quant-ph 2025-12 conditional novelty 6.0 of 10

    Two new classically verifiable benchmark scores, Clifford Volume and Free Fermion Volume, are defined, simulated under noise, and Clifford Volume is measured on the Quantinuum H2-1 device as 34 qubits.

  4. Quantum Computer Benchmarking: An Explorative Systematic Literature Review

    quant-ph 2025-09 conditional novelty 6.0 of 10

    A systematic review of 329 quantum benchmarking studies yields a stack-aligned taxonomy and definitions for hardware-, software-, and application-focused benchmarks.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · cited by 4 Pith papers

  1. [642]

    Estimation of Effective Temperatures in Quantum Annealers for Sampling Applications: A Case Study with Possible Applications in Deep Learning

    M. Benedetti, J. Realpe-Gómez, R. Biswas, and A. Perdomo-Ortiz, “Estimation of Effective Temperatures in Quantum Annealers for Sampling Applications: A Case Study with Possible Applications in Deep Learning”, Phys. Rev. A94, 022308 (2016) (Cited on page 101)

    work page 2016

  2. [643]

    Coherent Quantum Annealing in a Programmable 2,000 qubit Ising Chain

    A. D. King et al., “Coherent Quantum Annealing in a Programmable 2,000 qubit Ising Chain”, Nat. Phys.18, 1324 (2022) (Cited on page 101)

    work page 2022

  3. [644]

    Overcoming detection loss and noise in squeezing-based optical sensing

    G. Frascella, S. Agne, F. Y. Khalili, and M. V. Chekhova, “Overcoming detection loss and noise in squeezing-based optical sensing”, npj Quantum Inf.7, 72 (2021) (Cited on page 102)

    work page 2021

  4. [645]

    Single-mode squeezed-light generation and tomography with an integrated optical parametric oscillator

    T. Park, H. Stokowski, V. Ansari, S. Gyger, K. K. Multani, O. T. Celik, A. Y. Hwang, D. J. Dean, F. Mayor, T. P. McKenna, et al., “Single-mode squeezed-light generation and tomography with an integrated optical parametric oscillator”, Sci. Adv.10, eadl1814 (2024) (Cited on page 102)

    work page 2024

  5. [646]

    Measuring and exploiting the transmission matrix in optics

    S. Popoff, G. Lerosey, R. Carminati, M. Fink, A. Boccara, and S. Gigan, “Measuring and exploiting the transmission matrix in optics”, Quantum Electron. Laser Sci. Conf., QME6 (2010) (Cited on page 102)

    work page 2010

  6. [647]

    Complete characterization of quantum-optical processes

    M. Lobino, D. Korystov, C. Kupchak, E. Figueroa, B. C. Sanders, and A. Lvovsky, “Complete characterization of quantum-optical processes”, Science 322, 563 (2008) (Cited on page 102)

    work page 2008

  7. [648]

    Direct characterization of linear-optical networks

    S. Rahimi-Keshari, M. A. Broome, R. Fickler, A. Fedrizzi, T. C. Ralph, and A. G. White, “Direct characterization of linear-optical networks”, Opt. Express21, 13450 (2013) (Cited on page 102)

    work page 2013

  8. [649]

    Accurate and precise characterization of linear optical interferometers

    I. Dhand, A. Khalid, H. Lu, and B. C. Sanders, “Accurate and precise characterization of linear optical interferometers”, J. Opt. 18, 035204 (2016) (Cited on page 102)

    work page 2016

Show all 15 references
  1. [650]

    A versatile single-photon-based quantum computing platform

    N. Maring, A. Fyrillas, M. Pont, E. Ivanov, P. Stepanov, N. Margaria, W. Hease, A. Pishchagin, A. Lemaître, I. Sagnes, et al., “A versatile single-photon-based quantum computing platform”, Nat. Photonics18, 603 (2024) (Cited on page 103)

    2024

  2. [651]

    Classical Models May Be a Better Explanation of the Jiuzhang 1.0 Gaussian Boson Sampler than Its Targeted Squeezed Light Model

    J. Martínez-Cifuentes, K. M. Fonseca-Romero, and N. Quesada, “Classical Models May Be a Better Explanation of the Jiuzhang 1.0 Gaussian Boson Sampler than Its Targeted Squeezed Light Model”, Quantum7, 1076 (2023) (Cited on page 103)

    2023

  3. [652]

    Simulating Complex Networks in Phase Space: Gaussian Boson Sampling

    P. D. Drummond, B. Opanchuk, A. Dellios, and M. D. Reid, “Simulating Complex Networks in Phase Space: Gaussian Boson Sampling”, Phys. Rev. A105, 012427 (2022) (Cited on page 104)

    2022

  4. [653]

    Preparing random states and benchmarking with many-body quantum chaos

    J. Choi et al., “Preparing random states and benchmarking with many-body quantum chaos”, Nature613, 468 (2023) (Cited on page 106)

    2023

  5. [654]

    Benchmarking highly entangled states on a 60-atom analogue quantum simulator

    A. L. Shaw, Z. Chen, J. Choi, D. K. Mark, P. Scholl, R. Finkelstein, A. Elben, S. Choi, and M. Endres, “Benchmarking highly entangled states on a 60-atom analogue quantum simulator”, Nature628, 71 (2024) (Cited on pages 106, 107)

    2024

  6. [655]

    Multi-qubit entanglement and algorithms on a neutral-atom quantum computer

    T. M. Graham et al., “Multi-qubit entanglement and algorithms on a neutral-atom quantum computer”, Nature604, 457 (2022) (Cited on page 107)

    2022

  7. [656]

    Compiling Quantum Circuits for Dynamically Field-Programmable Neutral Atoms Array Processors

    D. B. Tan, D. Bluvstein, M. D. Lukin, and J. Cong, “Compiling Quantum Circuits for Dynamically Field-Programmable Neutral Atoms Array Processors”, Quantum8, 1281 (2024) (Cited on page 108). 124

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.