REVIEW 4 major objections 5 minor 27 references
On the Figures of Merit for Quantum Software Security: Toward a Benchmarking Rubric
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Quantum software security can be measured like performance through Security Figures of Merit and a rubric that aggregates them into a single Quantum Software Security Posture score.
desk verdict A useful taxonomy and a correct diagnosis of non-comparability, but the QSSP rubric's comparability is asserted rather than demonstrated—worth reviewing, not yet worth adopting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying objects are the S-FoMs themselves — scalar quantities that each measure one security characteristic or overhead under a declared adversary and a stated protocol, and that normalize to [0,1] — together with the rubric's three steps. Eq. (1) is a max-min normalization $\hat{m}_i = \min\{1, \max\{0, (m_i - m^\circ_i)/(m^\star_i - m^\circ_i)\}\}$ between declared reference points (weakest and saturation values), which absorbs orientation so lower-is-stronger metrics like DFC work. Eq. (2) forms the composite $\mathrm{QSSP} = \sum_i w_i \hat{m}_i$ with weights from a multi-criteria method, published with the score vector and geometric minimum $\prod_i \hat{m}_i^{w_i}$ as a conservative alternative. Security overhead $\widehat{SO}$ is not summed into QSSP but reported alongside, and $\mathrm{QSSP}_\lambda = \mathrm{QSSP}(1 - \lambda(1 - \widehat{SO}))$ exposes the trade-off. The attack-model declaration step is what makes each reported number meaningful.
What would settle it
Re-run the illustrative reanalysis with the paper's standardized TVD definition (splitting the summed count difference by twice the shots, so the value lies in [0,1]) and apply Eq. (1) with fixed reference points to the Dummy-CNOT and randomized-reversible-gate obfuscation tools; if the resulting normalized scores of the two tools are equal or the QSSP ordering reverses under small changes to the declared weights, the rubric's claim to restore comparability would be falsified.
Extended reading notes
Core claim
The central claim is that quantum software security can and should be treated as a measurable quantity on the same footing as performance. Concretely, the paper proposes that every relevant security property be captured by a Security Figure of Merit: a single quantity that measures one security characteristic or overhead, is computed by a stated protocol against one declared attack model, and normalizes to a common [0,1] scale where higher is stronger. The set is organized by ISO/IEC 25010 security sub-characteristics (confidentiality, integrity, authenticity, non-repudiation and accountability, resistance, plus a cross-cutting security-overhead row), mapped to the QaaS pipeline stages, and tagged Established, Adaptable, or Proposed depending on whether a quantum-specific measurement protocol exists. The rubric then normalizes via max-min scaling between declared reference points and aggregates the scores into a QSSP value by a weighted sum, with weights elicited by a multi-criteria method; the QSSP is published with its score vector and the geometric-mean minimum, and Security Overhead is kept separate so that security and cost can be read as a Pareto trade-off. The paper's reanalysis of the Dummy-CNOT and randomized-reversible-gate obfuscation techniques illustrates that their original TVD/DFC numbers are not comparable, and that the proposed normalization restores comparability.
Load-bearing premise
The rubric's comparability rests on the assumption that each security property can be reduced to one scalar measured against one declared attack model, and that benchmark builders can fix the normalization reference points consistently across tools.
Editorial extensions
If this is right
- A practitioner can state, with a shared definition and rubric, that tool A is more secure than tool B for a given quantum software stack and attack model.
- The maturity tags (E/A/P) make visible exactly which security properties are measurable today and where new quantum-specific protocols are urgently needed, notably the Proposed accountability metrics.
- Existing published obfuscation results (TVD, DFC) become comparable once reported under the standardized definition and normalization, so prior literature can be reanalyzed rather than discarded.
- Security overhead stays visible as a separate axis, so a tool that dominates on both security posture and overhead is preferred, analogous to the volumetric performance benchmark.
- The rubric can serve as a template for future quantum software security benchmarking, including a tool that computes established S-FoMs over Qiskit and Cirq programs.
Reading between the lines
- If benchmark builders adopt the proposed reference-point convention, the rubric could be applied retroactively to the existing obfuscation literature, producing a common table of security scores without new measurements.
- The S-FoM structure suggests a natural 'quantum security benchmark suite' analogous to Quantum Volume and CLOPS, in which vendors report a QSSP score alongside performance figures; whether regulators or cloud providers would require such reporting is an open question the paper leaves implicit.
- The geometric-mean minimum (conservative, non-compensatory) could become the default for security-critical deployments, where a weighted-sum average can hide a collapse in one property; the paper presents it as an alternative, not a recommendation.
- A testable extension is to check whether the [0,1] normalization of a lower-is-stronger metric like DFC, with reference points $m^\star=-1$ and $m^\circ=1$, actually preserves the ordering of obfuscation strengths across a wider library of circuits, since DFC's known range may not be fully covered.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that quantum software security currently lacks the standardized measurement infrastructure that performance benchmarking enjoys. It identifies a three-layer measurement gap (mature performance FoMs, emerging software-quality metrics, fragmented/absent security metrics) and proposes a structured set of Security Figures of Merit (S-FoMs) organized by ISO/IEC 25010 security sub-characteristics, QaaS pipeline stage, and measurement maturity (Established/Adaptable/Proposed). The paper then defines a benchmarking rubric with three steps: declare an attack model and measurement protocol, normalize each S-FoM to [0,1] via Eq. (1), and aggregate via Eq. (2) into a Quantum Software Security Posture (QSSP) score, with an optional security-overhead trade-off. An illustrative reanalysis of two published obfuscation techniques is presented to show that current TVD values are not comparable. The central claim is that this rubric turns non-comparable, scattered security numbers into an interpretable common scale.
Significance. If the proposed S-FoM taxonomy and rubric were fully instantiated, this would be a useful first step toward security-aware benchmarking for quantum software, filling a genuine gap. The paper's strengths include a clear three-layer gap analysis, a maturity-tagged S-FoM table that maps properties to pipeline stages, an explicit requirement that attack models be declared, and a well-posed critique of TVD definitional inconsistency in the obfuscation literature. The paper also makes sensible choices: publishing anchors and weights, reporting a score vector alongside QSSP, and considering a conservative geometric minimum. However, the central comparability claim is not demonstrated: Eq. (1) is only an affine rescaling, the illustrative reanalysis never computes a normalized score or QSSP, and most S-FoMs are labeled Adaptable or Proposed with no measurement protocol yet. The result is a defensible proposal whose load-bearing points need further work before the 'security measured like performance' claim is supported.
major comments (4)
- [Section IV, Eq. (1)] The normalization step is a purely affine rescaling of each S-FoM between two declared reference points m_min_i and m_star_i. It does not, by itself, make scores across metrics or across benchmark definitions commensurable: 0.5 on the normalized obfuscation score and 0.5 on the normalized outcome-verifiability score are not shown to correspond to equal security strength, and if each benchmark definition sets its own anchors, QSSP values from different benchmark definitions remain as incomparable as the raw TVD values the paper criticizes. The paper needs a protocol for anchor selection and calibration (for example, anchors tied to a stated adversary's capability or to physical saturation) and a demonstration that rankings are stable under reasonable anchor choices. As written, QSSP is a weighted sum of uncalibrated [0,1] rescalings, so the abstract's claim that the rubric 'turns non-comparable, scattered numbers into an interpretable comparison' is asserted rather than established.
- [Section V] The illustrative reanalysis does not instantiate the rubric. It restates that two published TVD values use incompatible definitions (one normalized by the number of shots and varying in [0,2], the other dividing by twice the shots and varying in [0,1]), but it never computes Eq. (1) scores for OS, FC, or RER, never selects reference points, never computes a confidentiality sub-score or QSSP, and never shows how the security-overhead trade-off would rank the two obfuscation techniques. The sentence 'S-FoMs restore the compatibility' is therefore a statement of intent, not a demonstrated result. At minimum, one worked example with declared anchors, weights, and an explicit comparison is needed to support the rubric's promised comparability.
- [Table I and Section III] The QSSP aggregation in Eq. (2) assumes that a chosen set of S-FoMs have measured values, but Table I labels most rows A (Adaptable) or P (Proposed), and Section III acknowledges that only confidentiality at S3 is well-instrumented. The paper does not specify how a benchmark should treat missing S-FoMs (drop them, impute a worst-case value, or report only the score vector), which leaves QSSP undefined for the full QaaS pipeline. Since the paper claims a multi-property security posture, a missing-data policy and a statement of which S-FoMs can currently be measured are needed before the rubric can be applied beyond the confidentiality examples.
- [Section IV, Eq. (2)] The text asserts that reported rankings must be demonstrated to be robust to plausible weight perturbations, but no perturbation analysis or worked example is provided, and AHP weight elicitation is only cited. Because QSSP is a compensatory weighted sum, weight choices can change rankings; the rubric should include at least a minimal sensitivity statement (for instance, report the range of rankings over a plausible weight interval) to support the reproducibility and fairness requirements named in Section IV.
minor comments (5)
- [Section V, paragraph 2] The phrase 'reports 1.92 and uniformly negative DFC' is ambiguous: 1.92 appears to be a TVD value under the [0,2] normalization, but the sentence structure reads as if DFC itself is 1.92, which would be outside the declared [-1,1] range. Please clarify which quantity 1.92 refers to.
- [Section II.A] The sentence 'This layer is proportionately standardized and supported by tools. As a result, provides us with a template...' has a missing grammatical subject in the second sentence; consider rewriting as 'As a result, this layer provides a template...'.
- [Section III] The sentence 'In addition, the table also mapped with the the QaaS pipeline stages of [13]' contains a duplicated 'the' and needs grammatical revision.
- [Section IV, Eq. (1)] The notation for the reference points is inconsistent: the equation uses m◦_i and m⋆_i, while the surrounding text refers to 'weakest value m_min_i' and 'strongest or saturation value m_star_i'. Please unify the notation.
- [Section VI] The phrase 'S-FoMs restore the compatibility' would read more clearly as 'S-FoMs can restore compatibility', since the restoration is contingent on following the rubric.
Circularity Check
No significant circularity: the rubric is definitional, and the paper's self-citations are contextual rather than load-bearing.
full rationale
The paper's claimed derivation chain is definitional rather than empirical. It proposes Security Figures of Merit, defines a normalization (Eq. 1) and a weighted aggregation (Eq. 2) into QSSP; no parameter is fitted to data and no external quantity is predicted from the rubric. The illustrative reanalysis in Section V does not compute any QSSP value; it only restates that published TVD values use incompatible definitions, which is a problem statement, not a circular derivation. The self-citations ([13] and [18]) provide the pipeline stage names and a prior classification as contextual framing; they are not used to justify the rubric's validity, and nothing in the rubric is argued to be uniquely forced by those citations. The only notable weakness is that comparability across tools depends on the reference points m_min_i and m_star_i being fixed by benchmark builders, which the paper explicitly acknowledges, and it does not supply a calibration protocol. That is an incompleteness or calibration gap, not a circular reduction: the rubric is honest about the anchors being part of the benchmark definition. Consequently there is no step in which a 'prediction' is equivalent to its input by construction.
Assumptions & free parameters
free parameters (2)
- Normalization reference points m_min_i and m_star_i per S-FoM =
Declared per benchmark; no values given
- Aggregation weights w_i =
To be elicited by AHP; no values given
assumptions (3)
- domain assumption ISO/IEC 25010:2023 security sub-characteristics are an appropriate taxonomy for quantum software security.
- domain assumption Each proposed S-FoM can eventually be measured under a stated protocol and attack model.
- domain assumption Min-max normalization with declared reference points produces comparable scores across tools and threat models.
Cite this review
Pith. "Pith review of On the Figures of Merit for Quantum Software Security: Toward a Benchmarking Rubric." pith.science (2026). https://pith.science/paper/WRUIWGPQ
@misc{pith2026260805831,
author = {Pith},
title = {Pith review of: On the Figures of Merit for Quantum Software Security: Toward a Benchmarking Rubric},
year = {2026},
howpublished = {\url{https://pith.science/paper/WRUIWGPQ}},
note = {Machine review of arXiv:2608.05831}
}
read the original abstract
Quantum software is increasingly provided through multi-tenant and cloud-based Quantum-as-a-Service (QaaS) stacks. A growing concern about the diverse attack vectors across the pipeline has been demonstrated in recent research. Yet the community has converged on three mature pillars: Scale (Qubit Count), Quality (Quantum Volume), and Speed (Circuit Layer Operations per Second (CLOPS)) for the merit performance figures. Moreover, it has also begun to define software-quality metrics. However, the security of quantum software remains largely unmeasured. A few quantitative security indicators, such as Total Variation Distance (TVD) and Degree of Functional Corruption (DFC), exist. Although they were introduced ad hoc for individual circuit obfuscation techniques, they are incompatible. We assert that the security of quantum software deserves the same attention as the performance: an explicit set of Security Figures of Merit (S-FoMs). The research of this paper is threefold: (i) characterizes a three-layer measurement gap, (ii) proposes a structured S-FoM set organized by ISO/IEC 25010 security sub-characteristics, QaaS pipeline mapping, and measurement maturity, and (iii) defines a benchmarking rubric that normalizes and aggregates S-FoMs into a combined Quantum Software Security Posture (QSSP) score. Additionally, an illustrative reanalysis of published obfuscation techniques has been presented. Our aim is a first step toward security-aware benchmarking of the Quantum Software Stack (QSS).
Figures
Reference graph
Works this paper leans on
-
[1]
Quantum Software Engineering: Roa dmap and Challenges Ahead,
J. M. Murillo et al., “Quantum Software Engineering: Roa dmap and Challenges Ahead,” ACM Transactions on Software Engineeri ng and Methodology, Jan. 2025, doi: https://doi.org/10.1145/37 12002
doi:10.1145/37 2025
-
[2]
V alidating quantum computers using randomized model circuits,
A. W. Cross, L. S. Bishop, S. Sheldon, P . D. Nation, and J. M . Gambetta, “V alidating quantum computers using randomized model circuits,” Physical Review A, vol. 100, no. 3, Sep. 2019, doi : https://doi.org/10.1103/physreva.100.032328
-
[3]
A. Wack et al., “Quality, Speed, and Scale: three key attr ibutes to measure the performance of near-term quantum computers,” a rXiv.org, Oct. 28, 2021. https://arxiv.org/abs/2110.14108
arXiv 2021
-
[4]
Some Size and Structure Metrics for Quantum Sof tware,
J. Zhao, “Some Size and Structure Metrics for Quantum Sof tware,” Ieee.org, 2021, doi: https://doi.org/10.1109/Q-SE52541 .2021.00012
-
[5]
Quantum Software Quality Metrics,
J. A. Cruz-Lemus, Mois´ es Rodr´ ıguez, Ra´ ul Barba-Rojas, and M. Piattini, “Quantum Software Quality Metrics,” pp. 125–142, Jan. 2024 , doi: https://doi.org/10.1007/978-3-031-64136-7 6
-
[6]
Systems and software engineering—SQuaRE—Produc t quality model,
ISO, “Systems and software engineering—SQuaRE—Produc t quality model,” ISO, 2023. https://www.iso.org/standard/78176. html
work page 2023
-
[7]
Short Paper: A Quantum Circuit Obfuscation Methodo logy for Security and Privacy,
A. Suresh, A. A. Saki, M. Alam, R. Onur Topaloglu, and S. Ghosh, “Short Paper: A Quantum Circuit Obfuscation Methodo logy for Security and Privacy,” Workshop on Hardware and Archite c- tural Support for Security and Privacy, pp. 1–5, Oct. 2021, d oi: https://doi.org/10.1145/3505253.3505260
-
[8]
Randomized Reversible Gate-Based O bfusca- tion for Secured Compilation of Quantum Circuit,
S. Das and S. Ghosh, “Randomized Reversible Gate-Based O bfusca- tion for Secured Compilation of Quantum Circuit,” arXiv.or g, 2023. https://arxiv.org/abs/2305.01133
arXiv 2023
Show all 27 references
-
[9]
A. A. Saki, A. Suresh, R. O. Topaloglu and S. Ghosh, ”Split Compila- tion for Security of Quantum Circuits,” 2021 IEEE/ACM Inter national Conference On Computer Aided Design (ICCAD), Munich, Germa ny, 2021, pp. 1-7, doi: 10.1109/ICCAD51958.2021.9643478
2021
-
[11]
QubitHamm er: Remotely Inducing Qubit State Change on Superconducting Quantum Com puters,
Y . Tan, N. Choudhury, K. Basu, and J. Szefer, “QubitHamm er: Remotely Inducing Qubit State Change on Superconducting Quantum Com puters,” arXiv.org, 2025. https://arxiv.org/abs/2504.07875 (acc essed May 26, 2026)
2025 arXiv
-
[12]
SW AP Attack: Stealthy Side-Channel Attack on Multi-Tenant Quan tum Cloud System,
L. Wei, S. Wang, S. Dutta, M. W. El, and A. Chattopadhyay, “SW AP Attack: Stealthy Side-Channel Attack on Multi-Tenant Quan tum Cloud System,” arXiv.org, 2025. https://arxiv.org/abs/2502.10115
2025 arXiv
-
[13]
An End-to-E nd Threat Model for the Quantum-as-a-Service Pipeline,
B. Rahman, M. Haghparast, and T. Mikkonen, “An End-to-E nd Threat Model for the Quantum-as-a-Service Pipeline,” 2026 IEEE In ternational Conference on Quantum Computing and Engineering (QCE), Acc epted, In-Press
2026
-
[14]
Inverse-Transpilation: Revers e-Engineering Quantum Compiler Optimization Passes from Circuit Snapsho ts,
S. Kundu and S. Ghosh, “Inverse-Transpilation: Revers e-Engineering Quantum Compiler Optimization Passes from Circuit Snapsho ts,” Pro- ceedings of the Great Lakes Symposium on VLSI 2025, pp. 273–2 77, Jun. 2025, doi: https://doi.org/10.1145/3716368.373529 8
2025
-
[15]
The Quantum Imitation Game: Reve rse Engineering of Quantum Machine Learning Models,
A. Ghosh and S. Ghosh, “The Quantum Imitation Game: Reve rse Engineering of Quantum Machine Learning Models,” Proceedi ngs of the 2024 Workshop on Attacks and Solutions in Hardware Secur ity, pp. 48–57, Nov. 2024, doi: https://doi.org/10.1145/3689939. 3695783
2024 doi
-
[16]
Trustworthy and reliable computing using untrusted and unreliable quantum hardware ,
S. Upadhyay and S. Ghosh, “Trustworthy and reliable computing using untrusted and unreliable quantum hardware ,” Frontiers in Computer Science, vol. 6, Aug. 2024, doi: https://doi.org/10.3389/fcomp.2024.1431788
2024
-
[17]
Quantu m Soft- ware Security Challenges Within Shared Quantum Computing E n- vironments,
S. Ovaskainen, M. Haghparast, and T. Mikkonen, “Quantu m Soft- ware Security Challenges Within Shared Quantum Computing E n- vironments,” 2025 IEEE International Conference on Quantu m Computing and Engineering (QCE), pp. 68–72, Aug. 2025, doi: https://doi.org/10.1109/qce65121...
2025
-
[18]
Classificat ion of security challenges and mitigation approaches in the quantum softwa re engineer- ing,
B. Rahman, M. Haghparast, and T. Mikkonen, “Classificat ion of security challenges and mitigation approaches in the quantum softwa re engineer- ing,” Journal of Systems and Software, vol. 238, p. 112884, A ug. 2026, doi: https://doi.org/10.1016/j.jss.2026.112884
2026
-
[19]
Accrediti ng outputs of noisy intermediate-scale quantum computing devices,
S. Ferracin, T. Kapourniotis, and A. Datta, “Accrediti ng outputs of noisy intermediate-scale quantum computing devices,” N ew Jour- nal of Physics, vol. 21, no. 11, p. 113038, Nov. 2019, doi: https://doi.org/10.1088/1367-2630/ab4fd6
2019 doi
-
[20]
Exploration of Power Sid e-Channel Vulnerabilities in Quantum Computer Controllers,
C. Xu, F. Erata, and J. Szefer, “Exploration of Power Sid e-Channel Vulnerabilities in Quantum Computer Controllers,” Procee dings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, Nov. 2023, doi: https://doi.org/10.1145/35769 15.3623118
2023 doi
-
[21]
Security Attacks Abusing Pulse-le vel Quantum Circuits,
C. Xu and J. Szefer, “Security Attacks Abusing Pulse-le vel Quantum Circuits,” 2025 IEEE Symposium on Security and Privacy (SP) , pp. 222–239, May 2025, doi: https://doi.org/10.1109/sp61157 .2025.00083
2025
-
[22]
Quan tum Physical Unclonable Functions: Possibilities and Impossibilities ,
M. Arapinis, M. Delavar, M. Doosti, and E. Kashefi, “Quan tum Physical Unclonable Functions: Possibilities and Impossibilities ,” Quantum, vol. 5, p. 475, Jun. 2021, doi: https://doi.org/10.22331/q-202 1-06-15-475
2021 doi
-
[23]
Analysis of c rosstalk in NISQ devices and security implications in multi-programmi ng regime,
Abdullah Ash-Saki, M. Alam, and S. Ghosh, “Analysis of c rosstalk in NISQ devices and security implications in multi-programmi ng regime,” Aug. 2020, doi: https://doi.org/10.1145/3370748.340657 0
2020
-
[24]
Co ntext Switching for Secure Multi-programming of Near-Term Quant um Com- puters,
A. Kumar, M. Wang, C. Liu, A. Li, P . J. Nair, and P . Das, “Co ntext Switching for Secure Multi-programming of Near-Term Quant um Com- puters,” arXiv.org, 2025. https://arxiv.org/abs/2504.07048
2025 arXiv
-
[25]
Decision making with the analytic hierarc hy process,
T. L. Saaty, “Decision making with the analytic hierarc hy process,” International Journal of Services Sciences, vol. 1, no. 1, p . 83, 2008, doi: https://doi.org/10.1504/ijssci.2008.017590
2008 arXiv
-
[26]
How to Build a Benchmark,
J. v. Kistowski, J. A. Arnold, K. Huppler, K.-D. Lange, J . L. Henning, and P . Cao, “How to Build a Benchmark,” Proceedings of the 6th ACM/SPEC International Conference on Performance Enginee ring, Jan. 2015, doi: https://doi.org/10.1145/2668930.2688819
2015
-
[27]
Handbook on constructing compo site indicators: methodology and user guide
Joint Research Centre. Handbook on constructing compo site indicators: methodology and user guide. OECD publishing, 2008
2008
-
[28]
A volumetric framewor k for quantum computer benchmarks,
R. Blume-Kohout and K. C. Y oung, “A volumetric framewor k for quantum computer benchmarks,” Quantum, vol. 4, p. 362, Nov. 2020, doi: 10.22331/q-2020-11-15-362
2020 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.