{"id":"4991a276-c950-4a40-99b3-3ec33835f75d","arxiv_id":"2608.10882","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A quantum-pattern detection pipeline is extended and evaluated on the Qrisp framework, reaching a micro-F1 of 0.712, then used to build composition graphs that separate direct pattern use from framework-internal pattern use.","lead":"This paper extends a tool that finds known quantum software patterns in open-source code, then evaluates it on a framework it was not trained on and builds call graphs of framework components. The graphs show which pattern uses are deliberate developer choices and which appear only because a framework calls its own routines internally, and the tool, knowledge base, and data are released for reuse.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline F1 gain rests on a consensus-negotiated Qrisp ground truth with Light's κ=0.316 and 18.2% exact-set agreement; plausible label changes could move micro-F1 across the 0.449 baseline.","rationale":"The reader's weakest assumption (manual KB labels) is real and is supported by the paper's own LLM re-check in Section 3.4, but it affects the F1 numbers only indirectly through the detector's pattern assignments. The Qrisp ground-truth labels are the direct measurement instrument for the headline 0.449-to-0.712 comparison. Section 3.6.1 reports agreement low enough that the 36-pair consensus is not a stable yardstick. This is an internally disclosed limitation, not an inconsistency: the paper reports κ honestly, but it does not quantify how much F1 depends on that ambiguity. A sensitivity analysis against individual rater labels would settle whether the vocabulary-expansion result is a real effect or an artifact of one consensus. The KB-label and composition-label concerns remain relevant, especially for RQ2/RQ3, but the Qrisp ground truth is the single most load-bearing assumption for the paper's main quantitative claim. I therefore keep the reader's CONDITIONAL verdict: no verdict change, but the condition should include demonstrating robustness of the F1 to rater variance.","tokens_in":26616,"tokens_out":8593,"duration_ms":87829,"concrete_test":"Recompute Table 2 micro-F1 separately against each of the three raters' pre-consensus label sets (the sets that produced κ=0.316) and report the range. If any single-rater micro-F1 is within 0.05 of the 0.449 no-expansion baseline, or if the across-rater range exceeds roughly 0.10, the claimed improvement is not robust to the documented ground-truth ambiguity. If per-rater labels are not archived, re-collect them by having the three raters relabel the 22 notebooks independently and repeat the scoring.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the reliability of the Qrisp ground truth behind the headline numbers (Section 3.6.1). The paper reports Light's κ=0.316 and an exact-set match rate of only 18.2% (4/22) among three raters. Disagreements include whether Block Encoding notebooks should be labeled Data Encoding/Matrix Encoding rather than Linear Combination of Unitaries, whether Domain Specific Application should replace specific sub-patterns in Shor/Sudoku/TSP, and which pattern set HHL.py instantiates. These ambiguities determine the 36 ground-truth pairs. The evaluation metric counts a prediction as correct when the pattern appears anywhere in the predicted set for that file, so recall (0.583) and micro-F1 (0.712) are directly sensitive to which patterns the consensus includes. With only 36 pairs and four patterns at zero true positives (Quantum Amplitude Estimation, VQA, Amplitude Amplification, Hamiltonian Simulation), adding or removing one ground-truth pair changes aggregate recall and F1 by several points. The paper's own disclosure of low agreement does not remove the threat: consensus after discussion can encode shared rater bias. Because the central quantitative claim is that vocabulary expansion raises F1 from 0.449 to 0.712, if a plausible alternative labeling yields an F1 below or close to 0.449, the RQ1 result is not established. This is a measurement-validity problem, not an internal inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper extends the qpa quantum-pattern mining pipeline and uses it to study pattern adoption and composition in open-source quantum software. It expands the knowledge base from 217 components across three frameworks to 286 components across five, adds two matching channels and a two-phase vocabulary-expansion step, and evaluates the extended detector on Qrisp, a framework excluded from the KB, reporting micro-F1 0.712 with expansion versus 0.449 without. It then builds projected call graphs for the five frameworks, defines a directness ratio to separate direct developer calls from framework-internal reach, and uses graph structure to explain pattern co-occurrence and compare the granularity at which frameworks expose patterns. The paper finds all 23 KB-represented patterns in 80 open-source projects and releases the pipeline, KB, datasets, and graph database.","tokens_in":26835,"tokens_out":6822,"duration_ms":68161,"significance":"If the evaluation held, the paper would be a useful step in empirical quantum software engineering: it provides the first quantitative detector evaluation on an out-of-distribution framework, introduces a composition analysis that distinguishes direct from indirect pattern use, and ships reproducible artifacts. Strengths include pinned framework versions, an open-source pipeline with Docker Compose setup, released ground-truth and usage datasets, and an LLM ensemble with cached outputs used to audit KB labels. The directness ratio and projected call graphs are simple, transferable concepts. However, the headline F1 result and several composition conclusions rest on measurement choices that need strengthening before the claims are fully supported.","major_comments":[{"comment":"The central RQ1 result is not robustly supported by the reported ground truth. The 36-pair gold set has Light's kappa of 0.316 and exact-set agreement of only 4/22 notebooks, and the disagreements listed (Block Encoding as LCU versus Data/Matrix Encoding, Domain Specific Application versus specific sub-patterns in Shor/Sudoku/TSP, and the pattern set of HHL.py) directly determine the true-positive counts for Grover, QPE, LCU, Amplitude Amplification, and Hamiltonian Simulation. Because the metric is micro-averaged over only 36 pairs and four patterns have zero true positives, a plausible alternative consensus label set could change aggregate recall by several points, potentially bringing the 0.712 micro-F1 close to or below the 0.449 baseline. The paper should report per-rater performance, confidence intervals, and a sensitivity analysis over alternative labelings, or rebuild the ground truth with more raters and stricter adjudication.","section":"Section 3.6.1, Table 2"},{"comment":"The evaluation protocol is transductive rather than a held-out generalization test. The \"two-phase pipeline\" constructs a dynamic KB from Qrisp's own source code before scanning Qrisp notebooks, so the improvement from 0.449 to 0.712 measures how much target-source vocabulary helps, not how well vocabulary expansion generalizes to an unseen framework's naming. The paper should state this explicitly and, for RQ1, either evaluate the expansion on a framework whose source was not used for the dynamic KB, or frame the claim only as adaptation to the target. This distinction matters because the abstract and RQ1 answer describe the result as a gain from \"vocabulary expansion\" on a framework \"not present in the knowledge base.\"","section":"Section 3.3 and Section 3.6.1"},{"comment":"The RQ3 cross-framework comparison is partially an artifact of graph scope. The paper states that Qiskit Algorithms and Qiskit Machine Learning appear flat because their calls to core components are recorded in a separate graph, and that Qiskit Algorithms has no hubs by construction. Comparing depth, hub share, and Grover decomposition across frameworks under these different scopes is therefore not apples-to-apples: the \"same pattern at different granularity\" claim in the RQ3 answer conflates implementation choices with package-boundary decisions. The analysis should either follow calls across package boundaries (listed as future work) or restrict the granularity comparison to frameworks with comparable graph scopes.","section":"Section 4.3.2, Table 8, Figure 7"},{"comment":"All downstream counts and composition measures inherit the manual single-label assignment of 286 components to patterns, and the paper itself demonstrates that these labels are fallible: the LLM re-check of the original 217-component KB found four Schmidt Decomposition entries that should have been assigned to three other patterns, and the authors then reran every analysis. This shows that label errors can propagate into all reported results. Because no independent audit is reported for the remaining entries, the paper should add a robustness analysis (for example, recomputing Table 1, Table 2, and the directness ratios under per-rater label sets or under a conservative perturbation of the KB) or provide a concrete argument for why such errors would not change the qualitative conclusions.","section":"Section 3.1.2 and Section 3.4"},{"comment":"The adoption claim that \"all 23 patterns occur in practice\" is based solely on qpa's detections on the 80-project corpus; the only manual validation is the 22-notebook Qrisp evaluation, which is not drawn from that corpus. Moreover, 440 of 611 detections (72%) come from dynamic-KB entries extracted from the selected projects themselves, so the scan can confirm patterns by matching a project's own vocabulary to itself. The paper should either sample and manually validate detections in the 80-project corpus or explicitly label the adoption counts as unvalidated estimates, and it should quantify how many of the 23 patterns are detected only through dynamic-KB entries.","section":"Section 3.5.1 and Section 3.5.3"}],"minor_comments":[{"comment":"The caption should explain that \"–\" means the metric is undefined because the pattern has no true positives (or no positive predictions), and it should state how the macro-average treats undefined per-pattern scores.","section":"Table 2 caption"},{"comment":"The sentence \"The ensemble labeled 214 components and reproduced the existing label for 160 of them\" should clarify the denominator: 160 of 214 classified components, not 160 of the original 217, and whether the four Schmidt Decomposition components were among the 214.","section":"Section 3.4"},{"comment":"The statement that \"stronger models would likely raise these numbers\" is unsupported speculation; it should be removed or explicitly marked as a conjecture.","section":"Section 3.6.2"},{"comment":"The units differ between the adoption analysis (notebooks/files in the 80-project corpus) and the composition analysis (usage files, including tests and framework-provided examples); Table 7 should state explicitly which file population is being counted.","section":"Section 4.1 and Table 7"},{"comment":"There are several typographical issues, including \"T able 1\", \"V ocabulary expansion\", and \"SW AP Test\" with a spurious space; these should be fixed in the final version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the artifacts are a genuine contribution. The main risk is not novelty but whether the headline evaluation can support the claim; the low inter-rater agreement is disclosed but not mitigated. I would ask for a sensitivity analysis or a re-annotation of the Qrisp ground truth before acceptance. The paper also depends heavily on the authors' prior MSR paper, so the incremental contribution should be assessed carefully against that baseline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a genuinely useful paper, and the part worth citing is not the F1 numbers; it's the composition graph idea. The directness ratio (whether a pattern's detections come from direct developer calls or from framework-internal calls) is a simple and clever way to reinterpret raw detection counts, and the co-occurrence analysis that separates forced, shared-building-block, and deliberate pairings is the strongest conceptual contribution. The authors ship the tool, KB, datasets, and graph database with a Docker Compose setup, and they disclose their limitations carefully. They even ran an LLM re-check of their own KB and corrected four mislabeled Schmidt Decomposition components. That kind of honesty is rare and should be credited.\n\nThe soft spots are real but mostly concentrated in RQ1. The Qrisp ground truth is small (36 pairs, 22 notebooks) and the raters agreed poorly (Light's kappa 0.316, exact set match 4/22). Since the metric checks whether a predicted pattern appears anywhere in the consensus set, the 0.712 micro-F1 is sensitive to which patterns survived the consensus discussion. The stress-test note is right: a plausible alternative labeling could put the expanded pipeline close to the 0.449 baseline, so the headline improvement is not robustly established. The paper does report macro-F1 (0.488) and discloses the inter-rater agreement, which mitigates the concern, but the absence of confidence intervals or a sensitivity analysis is a gap. Also, the Qrisp evaluation builds the dynamic KB from Qrisp's own source before scanning its notebooks: that's a legitimate adapted-pipeline test, but it's not a blind held-out evaluation, and the abstract doesn't say so. A related softer spot: the adoption claim that all 23 patterns occur in practice is tool-generated without independent manual validation on the analyzed projects. The manual single-label KB classification is the load-bearing assumption for both detection and composition, and the paper's own LLM check shows it's fallible.\n\nMy overall take: the composition analysis (RQ2/RQ3) is new conceptual work that survives these issues, and the artifacts make it reproducible. This deserves a serious refereeing round; I'd want the authors to add a sensitivity analysis or confidence intervals for the F1, do a small manual spot-check of detections on the adoption corpus, and state clearly that the Qrisp result is for the adapted pipeline. The paper is aimed at quantum software engineering researchers and tool builders; for that community it's a meaningful increment, not a breakthrough.\n\nRecommendation: send to peer review. If I were handling it, I'd ask for the F1 robustness analysis before publication, but the composition graphs alone justify referee time.","headline":"Composition graphs and the directness ratio are the real contribution; the Qrisp F1 claim is shakier than it looks but the paper is honest and reproducible.","tokens_in":27479,"tokens_out":2433,"would_cite":true,"duration_ms":22022,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Quantum-pattern counts mix developer calls with framework internals; the expanded detector reaches micro-F1 0.712 on a held-out framework.","keywords":["quantum software patterns","pattern mining","composition graphs","vocabulary expansion","knowledge base","LLM ensemble","Qrisp evaluation","quantum software engineering"],"falsifier":"Re-run the Qrisp evaluation after an independent source-code review of all 286 component-to-pattern assignments; if any corrected labels move components between patterns, the reported micro-F1 of 0.712 and every directness ratio change. A cheaper check is to run the same two-phase pipeline on a second held-out framework with a different naming convention and see whether the vocabulary-expansion step again raises recall from about 0.3 to above 0.5.","tokens_in":1691,"feed_emoji":"⚛️","tokens_out":4874,"duration_ms":107402,"temperature":0.7,"pith_summary":"Quantum software patterns are meant to be reusable building blocks, but before this paper there was little measurement of whether detection counts reflected what developers actually chose to call, and no accuracy check of the detector at all. The paper tries to establish two things: that a pattern-mining pipeline can generalize to a framework outside its knowledge base when it first learns that framework's vocabulary from its own source code, and that call graphs among framework components let an analyst separate deliberate developer calls from patterns that enter a program through framework-internal composition. On Qrisp, a framework deliberately excluded from the knowledge base, the expanded pipeline reaches micro-F1 0.712 (precision 0.913, recall 0.583) against 0.449 without vocabulary expansion. Applied to 80 open-source projects, the pipeline detects all 23 represented patterns and shows that counts concentrate in utility and variational patterns, while the composition graphs show that frameworks expose the same pattern at different granularities. A sympathetic reader would care because the result turns pattern adoption from a counting exercise into a structural analysis of how quantum frameworks are built.","feed_headline":"Quantum pattern detector reaches F1 0.712 on a held-out framework","feed_subtitle":"Vocabulary expansion nearly doubles recall on an unseen framework; call graphs separate developer calls from internals.","key_machinery":"Two mechanisms carry the argument. The first is two-phase vocabulary expansion: before scanning notebooks, qpa extracts non-private functions and classes from each project's source, associates them with patterns either by documentation similarity at a threshold of $\\geq 0.70$ to a seed component or by name matching against pattern names, and merges the results into a project-specific dynamic KB that is combined with the seed KB. This is what lets a detector built on Qiskit, PennyLane, and Classiq vocabulary recognize Qrisp's algorithm-centric names. The second is the projected composition graph, stored in a graph database, in which usage files point to KB components through CALLS edges and reach further components through USES edges; the directness ratio $\\rho(c)=\\mathrm{direct}(c)/\\mathrm{reach}(c)$ then quantifies whether a component is usually written by hand or pulled in internally.","core_discovery":"The paper's central claim, on its own terms: the qpa pipeline, extended with two additional matching channels and a two-phase vocabulary expansion, is the first version of the tool whose output is measured against a manual ground truth, and it scores micro-F1 0.712 on Qrisp tutorial notebooks, with the expansion step alone raising micro-F1 from 0.449 by lifting recall from 0.306 to 0.583. The companion conceptual claim is that composition graphs built from CALLS and USES edges among the 286 knowledge-base components show that a detection count is not a neutral measure: Qiskit's most common pattern is driven by synthesis routines no developer writes by hand, while PennyLane's counts mostly reflect direct developer choices, and the same pattern (Grover is the worked example) appears as a single block in some frameworks and as a chain of named primitives in others.","pith_inferences":["A natural next test, not run in the paper, is to hold out a second framework such as Cirq or TensorFlow Quantum and check whether the expansion step produces a comparable recall gain; if the gain is specific to Qrisp, the transfer claim is weaker.","The directness ratio could be adopted outside quantum software as a metric for framework transparency, since it measures how much of a library's apparent pattern usage is actually hidden plumbing.","Because the LLM ensemble disagreed with the manual knowledge base on roughly a quarter of components, an independent full re-labeling of the 286 components would likely shift some counts and ratios even if the headline evaluation stands.","The recurring QRAM/QROM disagreement noted in the paper suggests the 61-pattern catalog may need a dedicated addressable-quantum-memory pattern, and adding one would change the knowledge base and the detection results."],"forward_implications":["If the Qrisp result generalizes, pattern detection on a new quantum framework should start by building a dynamic vocabulary from that framework's own source, rather than relying only on a static knowledge base.","Raw pattern counts should be reported together with directness ratios; otherwise Qiskit-style frameworks, where low-level synthesis routines dominate counts without any developer call, will look artificially pattern-heavy.","The composition graphs make co-occurrence interpretable: a pair of patterns can co-occur because one is built from the other, because both share a lower-level routine, or because a developer combined them deliberately, so future adoption studies should distinguish these three cases.","The LLM ensemble's audit corrected four mislabeled Schmidt Decomposition components, so the released knowledge base and all results derived from it reflect the corrected 23-pattern set rather than the earlier 24-pattern set.","The release of the pipeline, knowledge base, and graphs means the Qrisp evaluation and the directness-ratio tables can be reproduced or checked without re-running the paid LLM service."],"supporting_citations":[{"why":"Supplies the original pattern catalog, knowledge-base structure, and basic detection pipeline that this paper extends.","marker":"[12]"},{"why":"Provides the PlanQK pattern definitions from which the 61-pattern catalog is derived.","marker":"[13]"},{"why":"Supplies the Qiskit framework components that form a large part of the seed knowledge base.","marker":"[17]"},{"why":"Supplies the PennyLane template components included in the seed knowledge base.","marker":"[15]"},{"why":"Supplies the Classiq components included in the seed knowledge base.","marker":"[16]"},{"why":"Supplies the Qiskit Machine Learning components added to the knowledge base in this work.","marker":"[18]"},{"why":"Provides the Qrisp framework used as the held-out evaluation target with a deliberately disjoint vocabulary.","marker":"[19]"},{"why":"Provides the all-mpnet-base-v2 embedding model that carries all four semantic matching channels.","marker":"[30]"}],"fun_headline_variants":["Vocabulary expansion lifts quantum pattern F1 to 0.712","Qpa pipeline: F1 0.712 with expansion, 0.449 without","Pattern detector hits 0.712 F1 via vocabulary expansion","Quantum composition graphs separate dev calls from internals","Expanded qpa mining pipeline scores 0.712 on new framework"],"cache_read_input_tokens":29440,"weakest_assumption_plain":"The load-bearing premise is that each of the 286 framework components was correctly assigned to exactly one of the 23 patterns, a manual judgment the paper itself shows to be fallible when its LLM audit found four components mislabeled as Schmidt Decomposition.","fun_headline_variants_meta":{"raw":{"variants":["Vocabulary expansion lifts quantum pattern F1 to 0.712","Qpa pipeline: F1 0.712 with expansion, 0.449 without","Pattern detector hits 0.712 F1 via vocabulary expansion","Quantum composition graphs separate dev calls from internals","Expanded qpa mining pipeline scores 0.712 on new framework"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000372,"raw_usage":{"total_tokens":2006,"prompt_tokens":982,"completion_tokens":1024,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":933}},"tokens_in":598,"tokens_out":1024,"duration_ms":7030,"temperature":1.0,"reasoning_tokens":933,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:03:50.583427+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Qrisp evaluation after an independent source-code review of all 286 component-to-pattern assignments; if any corrected labels move components between patterns, the reported micro-F1 of 0.712 and every directness ratio change. A cheaper check is to run the same two-phase pipeline on a second held-out framework with a different naming convention and see whether the vocabulary-expansion step again raises recall from about 0.3 to above 0.5.","supporting_citations":[{"cited_title":"In: Proceedings of the 23rd International Mining Software Repositories Conference","cited_arxiv_id":null,"evidence_quote":"Supplies the original pattern catalog, knowledge-base structure, and basic detection pipeline that this paper extends."},{"cited_title":"Original site offline; pattern list republished at https://qpa-quantum-patterns.web.app/","cited_arxiv_id":null,"evidence_quote":"Provides the PlanQK pattern definitions from which the 61-pattern catalog is derived."},{"cited_title":"HuggingFace (2021)","cited_arxiv_id":null,"evidence_quote":"Provides the all-mpnet-base-v2 embedding model that carries all four semantic matching channels."}],"review_version":1}