{"id":"c2a17ae2-0939-4215-b3ad-88aeea8d5d6b","arxiv_id":"2411.15963","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SelectQA, a QUBO-based quantum annealing approach, matches classical regression test selection on some programs, but its advantage over BootQA is weakened by an unfair problem formulation.","lead":"SelectQA, a quantum annealing method for regression test case selection, is tested against classical heuristics and the quantum baseline BootQA. It matches classical methods on some benchmarks, but its reported edge over BootQA comes from solving a different, unconstrained problem.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BootQA comparison is compromised because Eq. 11 is an unconstrained, separable linear objective trivially solved by thresholding, whereas BootQA solves a constrained optimization problem.","rationale":"The reader's weakest_assumption identifies exactly the issue I find load-bearing. Section 3.1.2 presents Eq. 11 as a two-objective formulation, but it omits the coverage penalty that the previous procedure introduced in Eqs. 7-10. With no constraints and no quadratic terms, Eq. 11 is a weighted-sum linear objective whose global optimum is obtained by an independent per-test threshold decision; no quantum annealing, and not even a hybrid solver, is required. BootQA is presented as a bootstrap-sampling solution to a test-case minimization problem, so it plausibly includes a coverage constraint or a cardinality objective that Eq. 11 lacks. Comparing a constrained optimizer against an unconstrained threshold rule cannot support the claim that SelectQA 'outperforms BootQA in effectiveness.' I also note that the algebra in Section 3.1.1 is internally inconsistent: Eq. 6 penalizes (x_i-1)^2 rather than (sum x_i -1)^2, and the expansion in Eqs. 8-10 does not follow from Eq. 7. That issue affects the classical three-objective comparison as well, but the BootQA comparison alone is sufficient to keep the current conditional verdict. The classical comparison against Additional Greedy and DIV-GA could still be a valid contribution if Eq. 10 is corrected and the released implementation matches the corrected model, so I do not recommend rejection; however, the authors should align the formulations and rerun the BootQA experiments on the same constrained task before Takeaway #3 is accepted.","tokens_in":15463,"tokens_out":7225,"duration_ms":73782,"concrete_test":"One decisive check: re-run SelectQA on PaintControl and GSDTSR using BootQA's actual QUBO, including whatever coverage, cardinality, or coupling constraints Eq. 11 omits. Independently inspect the online repository: if the submitted QUBO matches Eq. 11, verify that its output is identical to the threshold rule x_i = 1 iff alpha*cost_i < (1-alpha)*f_i computed classically; if the repository QUBO contains additional constraint terms, then Eq. 11 is not the model actually evaluated. If adding the missing constraint changes the dominance results in Table 7, the claimed BootQA superiority is not a valid effectiveness comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main quantum-vs-quantum effectiveness claim (Takeaway #3, Section 4.3.4) rests on Eq. 11: H = alpha * sum(x_i * cost_i) - (1-alpha) * sum(f_i * x_i). This is not a QUBO in any substantive sense: there are no quadratic terms, no coupling between test cases, and no coverage or cardinality constraint. Each variable can be optimized independently: set x_i = 1 exactly when alpha*cost_i < (1-alpha)*f_i. The hybrid solver therefore has no combinatorial structure to explore, and the reported 'optimal solution in each run' is simply a per-test threshold rule. BootQA, by contrast, is described in Section 2.2 as solving a test-suite optimization problem whose first purpose is minimizing the number of test cases while promoting other objectives, and Section 3.1 states that both formulations should implement a Minimum Vertex Cover problem. If BootQA's QUBO contains a coverage constraint, or any coupling/cardinality term that Eq. 11 omits, the two algorithms are not solving the same task: SelectQA is free to pick any subset above a threshold, so its dominance over BootQA in Section 4.3.4 is an artifact of the unconstrained formulation rather than evidence of quantum annealing effectiveness. Section 3.1.2 says the two-objective version is obtained 'without repeating the previous procedure,' but the previous procedure ended with the coverage penalty in Eq. 10, and dropping that term changes the problem class. This is the most load-bearing problem for the headline claim; the algebra inconsistencies in Section 3.1.1 are secondary but reinforce the need for a corrected, unified formulation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SelectQA, a reformulation of regression test case selection as a QUBO problem solved with D-Wave's hybrid quantum annealing service. SelectQA is evaluated in two settings: a three-objective version (execution cost, past fault coverage, statement coverage) compared against Additional Greedy and DIV-GA on four SIR programs, and a two-objective version (execution time, failure rate) compared against the quantum baseline BootQA on two industrial datasets. The authors report that SelectQA outperforms BootQA in effectiveness, is comparable to or better than the classical baselines in effectiveness, and has nearly constant execution time regardless of problem size.","tokens_in":15816,"tokens_out":6218,"duration_ms":57022,"significance":"If the results held, the paper would provide a useful empirical data point that hybrid quantum annealing can produce competitive test suite selections with constant wall-clock time over the tested sizes. The replication package, use of established benchmarks (SIR, PaintControl, GSDTSR), and statistical testing over ten runs are strengths. However, the central quantum-vs-quantum claim is compromised by a formulation mismatch: the two-objective SelectQA Hamiltonian is unconstrained and separable, while BootQA solves a constrained combinatorial problem. In addition, the three-objective formulation contains an incorrect coverage penalty and inconsistent algebra. These issues are load-bearing for the abstract's headline claims, although the underlying idea is potentially salvageable with a corrected formulation and re-run experiments.","major_comments":[{"comment":"The two-objective Hamiltonian in Eq. (11) has no quadratic terms and no constraints. It is equivalent to H = Σ_i x_i [α·cost(τ_i) − (1−α)·f_i], so each variable x_i can be optimized independently by setting it to 1 exactly when α·cost(τ_i) < (1−α)·f_i. The 'optimal solution in each run' reported in §4.3.4 is therefore a per-test threshold rule, not the result of a combinatorial optimization over coupled variables. BootQA, by contrast, is described in §2.2 as solving a test suite minimization problem with constraints and sampling subproblems. Since Eq. (11) drops the coverage/cardinality penalty that appears in Eq. (10), SelectQA and BootQA are not solving the same problem, and the dominance reported in Table 7 and Takeaway #3 is an artifact of the unconstrained formulation. The statement in §3.1 that both versions 'implement an algorithm that resolves the Minimum Vertex Cover problem' is contradicted by Eq. (11).","section":"§3.1.2, Eq. (11); §4.3.4"},{"comment":"The coverage constraint is not encoded correctly. The penalty term in Eq. (6), Σ_k Σ_{i∈T_k} (x_i − 1)^2, equals |T_k| − Σ_{i∈T_k} x_i for binary variables, so it is minimized by selecting all test cases that cover statement k, not by selecting at least one as required by Eq. (5). The correct penalty for 'at least one selected' would be (Σ_{i∈T_k} x_i − 1)^2 or an equivalent formulation. In addition, the algebra in Eqs. (8)–(9) is inconsistent: the expansion in Eq. (8) does not follow from Eq. (7), and Eq. (9) drops a constant and changes the linear coefficient. Since Eq. (10) is presented as the BQM actually submitted to the solver, the implemented Hamiltonian and the experiments need to be rechecked against the intended constraint.","section":"§3.1.1, Eqs. (6)–(9)"},{"comment":"The α parameter is tuned on the same datasets used for the evaluation ('validated following repeated trials'), and in the two-objective formulation α directly selects the threshold that defines 'optimality'. Because Eq. (11) is linear and separable, tuning α on the test data makes the claim that SelectQA 'always finds the optimal trade-off solution' circular. The authors should provide a validation protocol, a sensitivity analysis over α, or a held-out dataset before claiming that SelectQA dominates BootQA in effectiveness.","section":"§5 (Internal Validity); §4.3.4"}],"minor_comments":[{"comment":"There is a typo: 'traditionaltest suite optimization' should read 'traditional test suite optimization'.","section":"Abstract"},{"comment":"The row labeled 'Additional Method' is not defined in the experiment configuration in §4.2.2; either define it or remove it.","section":"Table 3"},{"comment":"The figure lacks axis labels and a legend, making it difficult to see which points correspond to which algorithm and which runs; the caption should specify the plotted quantities.","section":"Figure 3"},{"comment":"The Vargha-Delaney effect size is cited as reference [41], but the correct bibliographic entry appears to be [42]; the citation should be corrected.","section":"§5, Conclusion Validity"},{"comment":"The phrase 'large 0 effect sizes' appears to be a typo; it should likely read 'large effect sizes' or 'effect sizes of 0.0'.","section":"§4.3.5"},{"comment":"The phrase 'without repeating the previous procedure' is vague; the authors should state explicitly whether the coverage penalty from Eq. (10) is intentionally omitted and why.","section":"§3.1.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains a note indicating it has been accepted for publication and provides a DOI. If this version is under journal review, the formulation errors in Eqs. (6)–(11) and the resulting BootQA comparison should be addressed before publication; if it is already published, these issues warrant a correction or erratum."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The three-objective comparison is the real content here. SelectQA, formulated as a QUBO with a coverage penalty, is tested against Additional Greedy and DIV-GA on four SIR programs with ten runs, non-parametric tests, effect sizes, and shared code. That part is a legitimate, reproducible empirical study, and the result—competitive with classical heuristics on effectiveness, with constant wall-clock time because the D-Wave service overhead dominates—is worth knowing, though it is not a headline result.\n\nThe headline result against BootQA does not hold. Eq. 11, the two-objective Hamiltonian used for the PaintControl and GSDTSR comparisons, is linear and separable: each x_i can be set independently by comparing α·cost_i with (1−α)·f_i. There are no quadratic terms, no coverage constraint, no coupling. The hybrid solver is effectively executing a threshold rule. BootQA, as the paper itself describes, solves a constrained optimization problem with a coverage/minimization objective. The two are not solving the same task, so Table 7 and Takeaway #3 are artifacts of the unconstrained formulation. The claim in Section 3.1 that both versions implement Minimum Vertex Cover is contradicted by Eq. 11 itself. This is the load-bearing flaw.\n\nThere is also an algebra problem in Eqs. 8–9: the expansion of the penalty term is not what is written, and the binary simplification x_i^2 = x_i is not applied consistently. That is minor next to the Eq. 11 issue, but it needs fixing. The α parameter is tuned on the same datasets used for evaluation; the paper discloses this, but it should be treated as a validation threat, not a fixed configuration.\n\nI would send this to reviewers, but with the expectation of major revision. The classical comparison is worth keeping; the BootQA comparison needs to be redone with a true QUBO that includes the coverage or cardinality constraint, or the claims need to be limited to what the linear threshold rule can honestly support. I would not cite the BootQA result as it stands.","headline":"The three-objective TCS study is solid and reproducible, but the BootQA headline collapses because Eq. 11 is a separable linear threshold rule, not a real QUBO.","tokens_in":16316,"tokens_out":5310,"would_cite":false,"duration_ms":46407,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A QUBO reformulation of test case selection lets quantum annealing match classical methods and beat the quantum baseline BootQA.","keywords":["regression testing","test case selection","quantum annealing","QUBO","multi-objective optimization","hybrid quantum-classical solver","empirical software engineering"],"falsifier":"Take a dataset where the per-test failure-rate-to-cost ratio is anti-correlated with statement coverage, compute the exact optimum of Eq. (11) by sorting test cases on $f_i/\\mathrm{cost}(\\tau_i)$, and compare it with SelectQA's returned suite and with BootQA's coverage-constrained suite: if the threshold solution differs from SelectQA's output, the solver is not optimizing its own objective, and if it matches, the reported effectiveness advantage over BootQA is an artifact of comparing a trivially separable objective with a constrained problem.","tokens_in":15293,"feed_emoji":"⚛️","tokens_out":13362,"duration_ms":104403,"temperature":0.7,"pith_summary":"This paper sets out to show that regression test case selection can be reformulated as a quadratic unconstrained binary optimization (QUBO) problem that a quantum annealer solves well enough to be practically useful. The proposed method, SelectQA, encodes the trade-off between execution cost and fault detection as a weighted Hamiltonian and hands it to a hybrid quantum-classical solver, then builds a Pareto frontier from the returned test cases. The empirical claim is that SelectQA finds more non-dominated solutions than DIV-GA and Additional Greedy on two of four open-source programs, beats the quantum baseline BootQA in effectiveness on both industrial datasets in 18 of 20 runs, and takes about 2.9 seconds regardless of suite size. If this holds, testers get a single-weight, roughly fixed-time quantum option instead of expensive genetic search or the decomposition-parameter tuning that BootQA requires.","feed_headline":"Quantum annealer beats quantum method, matches classical selectors","feed_subtitle":"SelectQA runs in constant time on tested suites, offering a parameter-light quantum option for regression testing.","key_machinery":"The load-bearing object is the QUBO Hamiltonian $H = \\alpha \\sum_i x_i\\, \\mathrm{cost}(\\tau_i) - (1-\\alpha)\\sum_i e_i x_i + P\\sum_k (\\sum_{i\\in T_k} x_i - 1)^2$, with $e_i$ replaced by the failure rate $f_i$ in the two-objective version. The binary $x_i$ says whether test $i$ is selected; the first sum charges for execution cost, the second rewards past fault detection, and the penalty term forces every statement covered by the original suite to stay covered. This one expression converts the multi-objective test case selection problem into the single-objective format a quantum annealer can minimize, and it defines what the hybrid solver is actually optimizing; the paper's Pareto-frontier construction then runs on the selected test cases. The key move is that the objective is a weighted sum with penalty terms, so no explicit constraint-solving or decomposition loop is needed.","core_discovery":"The paper's central claim is that the way test case selection is encoded for the annealer matters more than the annealing hardware itself. SelectQA encodes the selection problem as a QUBO Hamiltonian whose linear terms price each test case by its normalized execution cost and failure-detection value, with coverage enforced through penalty terms, and the hybrid solver minimizes it directly. Against classical methods on the four open-source programs, SelectQA produces the largest number of non-dominated reference-frontier solutions on grep and sed, loses to DIV-GA on gzip, and loses to Additional Greedy on flex. Against BootQA on the two industrial datasets, SelectQA's returned suites dominated BootQA's in 18 of 20 runs, were never dominated, and had statistically significantly lower cost and higher failure rate. The paper also reports a practically constant total runtime near 2.9 seconds across all four programs and both datasets, which it credits to the hybrid solver rather than to any problem-specific decomposition.","pith_inferences":["Editorial inference: the two-objective QUBO (Eq. 11) separates per test case, so its optimum is a simple threshold on the ratio $f_i/\\mathrm{cost}(\\tau_i)$; if so, a classical sort would reproduce SelectQA's suites on the BootQA datasets, meaning the reported effectiveness win is not evidence of any quantum advantage.","Editorial inference: the constant 2.9-second runtime is likely the hybrid solver's fixed time budget rather than a scaling law of quantum annealing; separating QPU access time from total solver time on larger suites would test this.","Editorial inference: adding the coverage penalty from the three-objective version to the two-objective comparison would make the contest with BootQA a like-for-like constrained optimization and would clarify whether the method's value lies in the QUBO encoding or in the unconstrained reformulation."],"forward_implications":["SelectQA offers a single-weight ($\\alpha$) quantum selector whose wall-clock time stays near 2.9 seconds across the suite sizes studied, removing the need for the $(m,n)$ decomposition sweep BootQA requires.","On grep and sed, SelectQA contributed the largest share of non-dominated solutions to the reference frontier; on gzip, DIV-GA remained the best, so the paper does not claim one approach dominates everywhere.","Because SelectQA's Pareto frontier is built incrementally from annealing output, the method can be inserted into existing regression-testing pipelines that already construct frontiers from greedy or genetic search.","When wall-clock time is the binding constraint, SelectQA is the faster option than DIV-GA on all four programs, making quantum annealing a practical alternative for mid-size test suites."],"supporting_citations":[{"why":"Defines the multi-objective test case selection formulation and the Additional Greedy baseline used in the classical comparison.","marker":"[4]"},{"why":"Supplies DIV-GA, the diversity-guided genetic baseline, and the four-programs study context.","marker":"[8]"},{"why":"Presents BootQA, the quantum annealing baseline, its datasets, and the decomposition approach SelectQA is compared against.","marker":"[18]"},{"why":"Describes the hybrid quantum-classical solver service whose runtime metric underlies SelectQA's efficiency results.","marker":"[34]"},{"why":"Provides the penalty-based coverage constraint integration used to build the three-objective QUBO.","marker":"[35]"},{"why":"Gives the QUBO formulation and penalty-weight procedure used to turn the selection objectives into the Hamiltonian minimized by the solver.","marker":"[36]"}],"fun_headline_variants":["Rethinking test selection: QUBO encoding beats hardware","SelectQA: Quantum encoding outdoes prior quantum approach","Test suite selection: Quantum reformulation matches classics","Quantum annealing: How you encode matters more than the chip"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison with BootQA assumes that SelectQA's unconstrained two-objective QUBO (Eq. 11) is the same optimization task as BootQA's constrained minimization problem; if the two tasks are not equivalent, the paper's reported effectiveness dominance is an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Rethinking test selection: QUBO encoding beats hardware","SelectQA: Quantum encoding outdoes prior quantum approach","Test suite selection: Quantum reformulation matches classics","Quantum annealing: How you encode matters more than the chip"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1428,"prompt_tokens":916,"completion_tokens":512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":532,"tokens_out":512,"duration_ms":4991,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:41:18.163608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset where the per-test failure-rate-to-cost ratio is anti-correlated with statement coverage, compute the exact optimum of Eq. (11) by sorting test cases on $f_i/\\mathrm{cost}(\\tau_i)$, and compare it with SelectQA's returned suite and with BootQA's coverage-constrained suite: if the threshold solution differs from SelectQA's output, the solver is not optimizing its own objective, and if it matches, the reported effectiveness advantage over BootQA is an artifact of comparing a trivially separable objective with a constrained problem.","supporting_citations":[{"cited_title":"In: Proceedings of the 2007 International Symposium on Software Testing and Analysis, pp","cited_arxiv_id":null,"evidence_quote":"Defines the multi-objective test case selection formulation and the Additional Greedy baseline used in the classical comparison."},{"cited_title":"IEEE Transactions on Software Engineering 41(4), 358–383 (2014)","cited_arxiv_id":null,"evidence_quote":"Supplies DIV-GA, the diversity-guided genetic baseline, and the four-programs study context."},{"cited_title":"Test Case Minimization with Quantum Annealers","cited_arxiv_id":"2308.05505","evidence_quote":"Presents BootQA, the quantum annealing baseline, its datasets, and the decomposition approach SelectQA is compared against."},{"cited_title":"[Online] https: //www.dwavesys.com/media/m2xbmlhs/ 14-1048a-a d-wave hybrid solver service plus advantage technology update.pdf (Accessed on: 14-11-2024)","cited_arxiv_id":null,"evidence_quote":"Describes the hybrid quantum-classical solver service whose runtime metric underlies SelectQA's efficiency results."},{"cited_title":"Software Quality Journal, 1–30 (2023)","cited_arxiv_id":null,"evidence_quote":"Provides the penalty-based coverage constraint integration used to build the three-objective QUBO."},{"cited_title":"Gary Kochenberger: Quantum bridge analytics i: A tutorial on for- mulating and using qubo models","cited_arxiv_id":null,"evidence_quote":"Gives the QUBO formulation and penalty-weight procedure used to turn the selection objectives into the Hamiltonian minimized by the solver."}],"review_version":1}