{"id":"748cb4aa-7b1b-478e-8b2a-95fc8042862e","arxiv_id":"2602.14539","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 16-developer usability study found that both a local PQC library and an endpoint-based PQC API caused frequent key leaks, missing error checks, slow task completion, and no participant finished the full four-task secure-channel build.","lead":"This study tested how 16 developers coped with two post-quantum cryptography APIs, finding frequent key-handling mistakes and jargon-heavy documentation. It matters because NIST will deprecate RSA and ECDSA by 2035, and developer usability failures could undermine the security of the PQC migration.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Task-4 0% completion and declining completion rates are confounded by the two-hour session cap, so Table II does not cleanly measure API usability.","rationale":"The reader's verdict is CONDITIONAL, and I agree that is the right level. However, the reader's weakest_assumption focused on external validity—representativeness of the two APIs and artificiality of the setting. The more load-bearing concern is internal: the study's own pilot data and session procedure make the completion-rate metrics, especially Task 4's 0%, uninterpretable as measures of API usability. The paper reports that the pilot exceeded 90 minutes, and the main study still scheduled four tasks plus a questionnaire within two hours. Table V shows the first three tasks alone consumed over 105 minutes on average for the QuantCrypt group, so no participant could reasonably reach Task 4. This is not a generalizability caveat but a direct threat to one of the paper's strongest quantitative claims. I do not think this invalidates the qualitative CDF findings or the documentation-related observations, which are plausible and consistent with prior work. It does mean the quantitative task-completion evidence should be substantially de-emphasized or re-analyzed before the paper's headline claims are accepted. The reader already flagged a related issue ('task design that made the final task impossible to complete'), so my agreement is partial rather than full. The conditional verdict remains appropriate, pending the proposed re-analysis.","tokens_in":21672,"tokens_out":7998,"duration_ms":82708,"concrete_test":"Re-analyze existing session recordings/logs to compute, for each participant, the time at which Task 4 instructions were first presented and the time remaining until the two-hour cap; if the median remaining time is less than the 15–20 minutes specified for Task 4, the 0% completion rate is a time-budget artifact. Additionally, re-report Task 1–3 completion rates after excluding participants who terminated because the moderator asked if they wished to continue.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is internal validity: the headline completion-rate results are confounded by a fixed time budget. The pilot (§III-D) found average task duration exceeded 90 minutes, yet the main study kept four sequential tasks (15–20 minutes each), plus a briefing and a 20–30 minute post-task questionnaire, all under a two-hour session cap with an explicit 'do you wish to continue?' prompt at the two-hour mark. Raw times in Table V imply mean cumulative Task 1+2+3 time of about 106 minutes for QuantCrypt and 114 minutes for PQ-Sandbox, before the questionnaire. Most participants therefore never had a real opportunity to attempt Task 4, and the reported 0% Task 4 completion rate—and plausibly part of the 50–87.5% completion rates for Tasks 2 and 3—is an artifact of session length, not API usability. The paper's strongest evidence-based claim explicitly cites 'no participant finished the four-task secure-channel construction' as evidence of PQC difficulty, but this is not a valid usability outcome under these time constraints. This also undermines RQ2 comparisons, because completion-time means are computed on truncated task subsets and are not comparable across groups.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a between-subjects usability study of two post-quantum cryptography (PQC) APIs: QuantCrypt, a local Python library, and PQ-Sandbox, an endpoint-based API. Sixteen developers (eight per API) performed four sequential tasks—KEM key establishment, symmetric encryption, DSA-authenticated handshake, and DSA-signed message exchange—while thinking aloud under remote observation. The authors report task completion rates and times, analyze final code for security-relevant flaws, and map qualitative questionnaire responses to an extended Cognitive Dimensions Framework. They claim that PQC API usability has not been systematically studied before, that developers encounter pervasive security-relevant misuses (e.g., 100% omitted KEM error handling and key destruction, several shared-secret leaks), and that no participant completed all four tasks. The paper recommends secure-by-default high-level functions, better documentation, and NIST-aligned terminology.","tokens_in":21881,"tokens_out":5154,"duration_ms":53772,"significance":"The topic is timely and important: PQC migration is proceeding rapidly, and developer-facing API usability is a likely source of deployment failures. The study has real methodological strengths: standardized skeleton code, think-aloud sessions with screen recordings, verbatim participant quotes, dual-coder thematic analysis, and full raw time data in an appendix. If the empirical claims were robust, the paper would be a valuable contribution to usable security and PQC adoption. However, the central quantitative claims are only partially supported. The fixed two-hour session cap confounds the headline completion-rate results, the sample is small and underpowered for the t-tests used, and one of the two compared APIs is a vendor research prototype provided by a co-affiliated company. These issues limit internal and external validity. The qualitative insights about documentation, terminology, abstraction, and developer mental models remain useful and could support a defensible revised paper.","major_comments":[{"comment":"The completion-rate results are confounded by the fixed two-hour session cap. The pilot (§III-D) found average task duration exceeded 90 minutes, yet the main study kept four 15–20 minute tasks plus a briefing and a 20–30 minute questionnaire under a two-hour cap, with an explicit 'do you wish to continue?' prompt. Raw times in Table V imply cumulative T1+T2+T3 times at or above 120 minutes for several participants (e.g., QC P16: 57+54+30=141 minutes) before questionnaire time. Therefore, the 0% Task 4 completion rate—and part of the 50–87.5% rates for Tasks 2 and 3—reflect session length, not API usability. This undermines the strongest evidence-based claim in the abstract and the RQ2 completion-time comparisons, since means are computed on truncated, non-random subsets (T2 N=7 vs 7; T3 N=5 vs 4). Please re-analyze using time-to-task-attempt data, report completion as 'not completed wit","section":"§III-D, Table II, Table V"},{"comment":"The inferential statistics are not robust enough for the strength of the claims. The paper reports unpaired t-tests on subsets of 8, 7, and 5/4 participants without a pre-registered analysis, power analysis, or correction for multiple comparisons. The language 'clear and statistically significant' (p=0.0079, p=0.0015) overstates confidence: with N=16 and missing data, the meaningful signal is the descriptive error patterns in Table III, not the p-values. Please re-frame the quantitative comparisons as exploratory and report effect sizes, bootstrapped confidence intervals, or nonparametric tests. The paper's own limitation section (VI) acknowledges the small sample but does not address the missing-data structure introduced by the time cap.","section":"§IV-B"},{"comment":"The comparison underpinning RQ2 is not a clean 'endpoint-based PQC API versus local library' comparison. PQ-Sandbox is described as a research prototype from ExeQuantum, 'intentionally simplified for experimentation,' and the acknowledgments thank Samuel Tseitkin and ExeQuantum for technical support with the APIs and Sandbox; Tseitkin is also an author. This conflates API architecture with prototype maturity and creates an undisclosed affiliation with one of the two evaluated systems. Results on PQ-Sandbox may reflect prototype quality rather than endpoint-based PQC APIs generally, and the Task 1/Task 2 time differences may be due to design choices of a vendor prototype. Please reframe RQ2 as comparing two specific implementations, disclose the affiliation in the methodology, and temper generalizability claims.","section":"§III-A, Acknowledgment"},{"comment":"The paper's scope claims exceed its design. The abstract states that 'the usability of PQC APIs has not been systematically studied' and the conclusion uses ecosystem-level language, but the evidence is a single study of two APIs with 16 participants. Section VI correctly limits 'quantitative generalizability,' yet the framing in the abstract and conclusion still suggests broad conclusions about the entire PQC API ecosystem. Please align the claims with the study design: the paper can legitimately report observed usability barriers and security-relevant mistakes in these two APIs and should not present them as established properties of PQC APIs in general.","section":"Abstract, §VII"}],"minor_comments":[{"comment":"The appendix heading 'TASKPEFORMANCERAWDATA' should be 'TASK PERFORMANCE RAW DATA.'","section":"Appendix C heading"},{"comment":"Typographical and grammatical issues: 'you need to runt the server' should be 'run'; 'what you read and work could be captured' is awkward. A light language edit is recommended throughout.","section":"Appendix B"},{"comment":"The legend is difficult to parse because the same symbols (●, ✗) are used for leak detection and missing handling under different columns. Consider separate sub-headers and explicit column meanings to avoid ambiguity.","section":"Table III"},{"comment":"There is a grammatical issue in the sentence 'Although time constraints contributed to this behavior, but unclear documentation...'—the 'although' and 'but' are redundant.","section":"§V-A"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper has a genuine qualitative contribution, but the quantitative claims and the RQ2 comparison need substantial revision before publication. The time-budget confound is severe enough that Table II should not be published as-is. I would also ask the authors to add a conflict-of-interest/funding statement; the involvement of an author affiliated with ExeQuantum in providing and supporting PQ-Sandbox is a governance issue the journal should flag. This is fixable with reframing and re-analysis, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the new thing: this is the first systematic usability study of PQC APIs that I know of, and it is a genuine empirical contribution. The think-aloud data, screen recordings, verbatim quotes, and the CDF-based analysis give a credible picture of where non-specialists stumble: low-level abstraction, jargon-heavy docs, unclear key-handling workflow, and missing testability guidance. The Table III error patterns (no error handling, no key destruction, shared-secret leaks) are concrete and useful, and they align with decades of classical crypto-API usability findings. That part is worth having.\n\nThe quantitative core, though, is weaker than the narrative. The 0% Task 4 completion rate is not a usability result: the pilot reported average task time over 90 minutes, yet the main study squeezed four tasks plus a 20-30 minute questionnaire into a two-hour cap. The raw times in Table V show cumulative T1–T3 around 106–114 minutes, so participants simply ran out of clock. The same cap likely deflated some Task 2 and Task 3 completion rates. The paper's strongest claim — \"no participant finished the four-task secure-channel construction\" — is therefore confounded and should not be cited as evidence of PQC difficulty. That stress-test note is right.\n\nThe statistical comparisons also need work. With N=8 per group and per-task subsets as small as 4–5, the t-tests are underpowered, and no correction for multiple comparisons is applied. The observed differences are large enough that they may survive, but the authors need to report effect sizes, justify the tests, and present these as exploratory.\n\nThere is also an unaddressed conflict of interest: two authors are from ExeQuantum, and PQ-Sandbox is their own research prototype. Acknowledgement of technical support does not deal with the possibility of bias, especially since the prototype is \"intentionally simplified.\" The comparison is not apples-to-apples. The authors should either include a third-party API, or transparently discuss how their affiliation shaped the selection and analysis.\n\nOne more thing: the paper calls \"failure to add error handling\" and \"failure to destroy keys\" pervasive, but the tasks never asked participants to do those things. That is a legitimate usability observation about API guidance, but it is not a participant failure. The language should be calibrated.\n\nBottom line: the qualitative findings and the raw data are worth peer review. The quantitative claims need to be reframed, the COI addressed, and the statistics fixed. I'd send it to review as a conditional acceptance, not desk-reject. It's the first dataset of its kind, and it will be cited by people building PQC guidance.","headline":"First systematic usability study of PQC APIs, with real qualitative value, but the headline completion-rate finding is confounded by the session length and the API comparison has an unresolved conflict of interest.","tokens_in":22421,"tokens_out":3225,"would_cite":true,"duration_ms":33291,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A usability study of two post-quantum cryptography APIs finds that every participant omitted error handling and key destruction, with multiple shared-secret leaks.","keywords":["post-quantum cryptography","PQC API usability","developer study","KEM","ML-KEM","ML-DSA","think-aloud protocol","secure-by-default design"],"falsifier":"Ask a fresh cohort of, say, 40 professional developers to implement the same secure channel using a widely used production PQC library (or the same two APIs) with their normal documentation and usual search/AI tools, then audit for the seven error classes in Table III; if the rates of missing error handling and key destruction drop substantially below 100%, or if shared-secret leaks vanish, the claim that PQC APIs are intrinsically unusable for generalists would be falsified.","tokens_in":21514,"feed_emoji":"🔐","tokens_out":4332,"duration_ms":41827,"temperature":0.7,"pith_summary":"This paper argues that the usability of post-quantum cryptography (PQC) APIs is an unexamined and serious barrier to secure migration. In a controlled study, 16 developers implemented a KEM/DSA secure channel using two different PQC APIs with minimal onboarding; every participant failed to add error handling for encapsulation/decapsulation and none explicitly destroyed keys, while several leaked shared secrets. The authors identify cognitive friction points—low-level granularity, jargon-heavy documentation, hidden workflow sequencing—and trace them to API design choices. If correct, the findings mean that shipping NIST-standard algorithms inside an API is not enough: without secure-by-default design and developer-centered documentation, PQC adoption will repeat the misuse failures of classical crypto APIs.","feed_headline":"PQC APIs fail developers: 100% skipped error handling, keys leaked","feed_subtitle":"A head-to-head test of two post-quantum crypto libraries shows why NIST-standard algorithms alone won't stop insecure apps.","key_machinery":"The carrying mechanism is the controlled usability experiment combining think-aloud observation with the Cognitive Dimensions Framework (a taxonomy for rating how API design affects comprehension). Participants were asked to implement a KEM/DEM hybrid encryption channel and a digital-signature handshake using two contrasting PQC APIs—one a local library, one an endpoint service—and their code was audited for security-relevant errors. The framework maps observed friction to dimensions such as abstraction level, penetrability, and error-proneness, letting the authors tie specific API design choices (e.g., exposing KDF separately, returning 32-byte vs 64-byte keys, requiring auth headers) to sp","core_discovery":"The paper's central discovery is that non-specialist developers, when given PQC APIs with minimal onboarding, systematically produce insecure implementations: 100% of participants omitted error handling for KEM wrap/unwrap and explicit key destruction, 4 of 16 leaked shared secrets, and no one completed all four tasks in a secure-channel construction. The authors attribute these failures not to individual incompetence but to structural API usability issues—excessively low-level abstraction, documentation written for cryptography-aware readers, non-intuitive naming, and unstated operation sequencing—and support this with think-aloud transcripts and a Cognitive Dimensions analysis. The result","pith_inferences":["A direct testable extension: build a 'secure channel in one call' function implementing KEM, KDF, AES, and DSA internally, and run the same task with the same population; the paper predicts misuse rates would drop sharply for non-experts.","The 100% missing error-handling figure may partly reflect Python's exception style and the skeleton code's structure, so the same API in a language with explicit status codes could produce different—though not necessarily better—outcomes.","Because the study compared only two APIs, the specific rates are less robust than the qualitative pattern; similar cognitive frictions likely appear in other PQC libraries, but confirming this requires a broader multi-API study.","The performance trade-off between the two APIs (faster KEM setup vs faster symmetric integration) suggests that PQC API design is a resource-allocation problem; future APIs could offer profiles for different developer expertise levels rather than a single abstraction."],"forward_implications":["If PQC API usability is as poor as observed, the 2035 deprecation of RSA/ECDSA will push developers toward insecure integrations unless libraries add higher-level, secure-by-default entry points.","APIs that internally apply the KDF and return a ready-to-use symmetric key reduce one class of errors but introduce others, suggesting the optimal design may be a layered API with both high-level and low-level paths.","Documentation that shows complete, runnable client-server workflows—not just isolated function calls—could reduce trial-and-error assembly and the resulting misconfigurations.","Standardizing on NIST's encapsulation/decapsulation terminology would remove naming confusion that costs developers time and leads to key-confusion errors.","The absence of testing guidance correlates with untested crypto code; built-in test helpers or checkable error states would raise the chance that developers verify their implementations."],"fun_headline_variants":["PQC APIs: 100% of devs skipped error handling, 25% leaked keys","Quantum-resistant APIs fail usability: every developer missed key steps","Non-experts can't use PQC APIs: 100% error omission, keys exposed","Post-quantum APIs too hard: all 16 devs failed security basics","PQC API trial: 4 of 16 devs leaked secrets, 100% skipped checks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim assumes that the two APIs and the minimal-onboarding test setting are representative of how PQC APIs are actually used; if production PQC libraries come with better onboarding or the participants were atypical, the near-universal failure rates could be an artifact of the experiment rather than a property of PQC APIs.","fun_headline_variants_meta":{"raw":{"variants":["PQC APIs: 100% of devs skipped error handling, 25% leaked keys","Quantum-resistant APIs fail usability: every developer missed key steps","Non-experts can't use PQC APIs: 100% error omission, keys exposed","Post-quantum APIs too hard: all 16 devs failed security basics","PQC API trial: 4 of 16 devs leaked secrets, 100% skipped checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000715,"raw_usage":{"total_tokens":3060,"prompt_tokens":762,"completion_tokens":2298,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":2202}},"tokens_in":506,"tokens_out":2298,"duration_ms":16707,"temperature":1.0,"reasoning_tokens":2202,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:07:44.867301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a fresh cohort of, say, 40 professional developers to implement the same secure channel using a widely used production PQC library (or the same two APIs) with their normal documentation and usual search/AI tools, then audit for the seven error classes in Table III; if the rates of missing error handling and key destruction drop substantially below 100%, or if shared-secret leaks vanish, the claim that PQC APIs are intrinsically unusable for generalists would be falsified.","supporting_citations":[],"review_version":1}