{"id":"5aa2e0b4-4e25-42cb-aabc-c40561789f4b","arxiv_id":"2611.17003","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Three NIST-referenced crypto agility maturity models fail to provide intersubjectively verifiable criteria or prescriptive improvement guidance, undermining their suitability for governance and procurement.","lead":"This paper evaluates three industry-driven crypto agility maturity models referenced by NIST against established design principles for maturity models. It finds that all models systematically lack measurable, reproducible assessment criteria and integrated guidance for improving crypto agility, which limits their usefulness for governance and procurement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The evaluation framework (Pöppelbuß & Röglinger DPs) is applied as if its own descriptive and prescriptive principles are universally applicable, but the paper does not test whether the three industry artifacts were intended as full maturity models; if they were not, the central finding of…","rationale":"The reader's weakest_assumption correctly identifies that the paper relies on Pöppelbuß & Röglinger's DPs as the appropriate framework. My concern sharpens this: the paper applies the full DP set (including descriptive and prescriptive) to artifacts that, by the paper's own admission in §5, do not claim to be maturity models. The reader's verdict (ACCEPT, high confidence) does not flag this as a load-bearing issue. I argue it is: if the artifacts are not maturity models, then failing maturity-model DPs is not a valid critique. The paper's strongest claim—systematic failure on descriptive/prescriptive principles—would be a category error. The concrete test would settle this by checking the artifacts' own stated purposes. If none claims both intersubjectively verifiable criteria and integrated improvement guidance, the paper's central finding is misleading. I recommend CONDITIONAL: accept only if the artifacts' stated purposes are shown to include the descriptive and prescriptive goals that the DPs test. The paper's own §5 language ('none explicitly positions itself as a formal maturity model') already undermines the unconditional version of the claim.","tokens_in":17106,"tokens_out":780,"duration_ms":7230,"concrete_test":"For each of the three artifacts (FS-ISAC, ATIS, Deloitte), extract the authors' own stated purpose from the document's title, abstract, and introduction. Then check whether that stated purpose includes (a) providing intersubjectively verifiable assessment criteria and (b) providing integrated improvement guidance tied to maturity levels. If none of the three documents claims both (a) and (b), then the paper's application of DPs 2.x and 3.x is a category mismatch, and the central finding should be reinterpreted as 'these artifacts do not meet criteria they never aimed to meet.'","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim—that the three models systematically fail to provide intersubjectively verifiable criteria and prescriptive guidance—depends on treating FS-ISAC, ATIS, and Deloitte as maturity models subject to the full set of Pöppelbuß & Röglinger design principles. The authors acknowledge in §5 that 'none of the three models explicitly positions itself as a formal maturity model' and that ATIS 'would require significant additions or modifications to meet the strict definition of a maturity model.' Yet the evaluation (§4) applies the descriptive and prescriptive DPs as if the artifacts were intended to be full maturity models. If the artifacts are better understood as a sector-specific extension (FS-ISAC), a KPI framework (ATIS), and a CSF community profile (Deloitte), then failing DPs 2.x and 3.x is not a design flaw but a mismatch between the evaluation lens and the artifact's purpose. The paper's strongest result—systematic failure on descriptive/prescriptive DPs—would then be an artifact of applying an inappropriate benchmark, not a finding about the models' quality for their intended use. The paper does not establish that the artifacts' authors claimed descriptive or prescriptive maturity-model status; it infers this from NIST's citation and from broad statements about 'helping to evaluate/improve.' This is a load-bearing assumption that, if wrong, collapses the headline conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper evaluates three industry-driven crypto agility maturity models referenced by NIST against established design principles for maturity models (Pöppelbuß and Röglinger). It finds that while the models meet basic design principles to varying degrees (reflecting their intended purposes), they systematically fail to provide intersubjectively verifiable assessment criteria (descriptive principles) and integrated guidance for improving crypto agility (prescriptive principles). The paper offers recommendations for future model developers and guidance for practitioners selecting among the models.","tokens_in":17534,"tokens_out":1711,"duration_ms":11679,"significance":"The paper addresses a timely and practically important topic—crypto agility maturity models—given the ongoing migration to post-quantum cryptography and NIST's influential role. The systematic evaluation using established design principles is methodologically sound and provides a clear, structured comparison. The paper's main strength is its transparency: it documents per-principle verdicts, acknowledges limitations (subjective interpretation, lack of empirical application), and does not overclaim. The findings are useful for practitioners and model developers, and the recommendations are concrete. The paper does not introduce new models or data but provides a valuable meta-evaluation that fills a gap in the literature.","major_comments":[{"comment":"The paper's central claim—that the three models systematically fail to provide intersubjectively verifiable criteria and prescriptive guidance—depends on treating FS-ISAC, ATIS, and Deloitte as maturity models subject to the full set of Pöppelbuß & Röglinger design principles. The authors acknowledge in §5 that 'none of the three models explicitly positions itself as a formal maturity model' and that ATIS 'would require significant additions or modifications to meet the strict definition of a maturity model.' Yet the evaluation (§4) applies the descriptive and prescriptive DPs as if the artifacts were intended to be full maturity models. If the artifacts are better understood as a sector-specific extension (FS-ISAC), a KPI framework (ATIS), and a CSF community profile (Deloitte), then failing DPs 2.x and 3.x is not necessarily a design flaw but a mismatch between the evaluation lens and the artifact's purpose. The paper does not establish that the artifacts' authors claimed descriptive or prescriptive maturity-model status; it infers this from NIST's citation and from broad statements about 'helping to evaluate/improve.' This is a load-bearing assumption that, if wrong, weakens the headline conclusion. The authors should either (a) provide stronger evidence that the artifacts are intended as maturity models in the sense of Pöppelbuß & Röglinger, or (b) reframe the findings as an assessment of how well these artifacts function as maturity models when interpreted as such, acknowledging that this may not align with their original design goals.","section":"§3.1, §5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The evaluation of ATIS on DP 2.1 (intersubjectively verifiable criteria) is rated as '∼' (partially fulfilled). The authors note that only 5 of 23 metrics provide explicit transformation to the 1–10 scale, and the rest rely on undefined judgment. However, the paper does not discuss whether the 5 metrics that are verifiable are sufficient for a meaningful assessment of crypto agility maturity. A partial fulfillment rating may overstate ATIS's descriptive value if the verifiable metrics cover only a narrow subset of the domain. The authors should clarify what 'partially fulfilled' means in practice and whether the verifiable metrics are representative or merely a small fraction that does not support a holistic assessment.","section":"§4.2, Table 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The authors acknowledge that their evaluation is subjective and that they are not the primary target group. They also decided not to complement the analysis with empirical application, arguing that the lack of intersubjective verifiability would make such application reproduce the same findings. While this reasoning is logical, it also means the paper's conclusions about practical applicability remain untested. The authors should discuss whether a small-scale empirical test (e.g., applying one model to a hypothetical organization) could have provided additional insights or validation, even if limited. As it stands, the paper's recommendations for practitioners (e.g., 'choose by purpose of use and background') are based on the design-principle analysis alone, without evidence that the models work as intended in practice.","section":"§5.3","authors_acknowledge":"Finally, we decided not to complement our design-principle-based analysis with empirical application of the models.","authors_acknowledge_p":0.89}],"minor_comments":[{"comment":"The introduction mentions that NIST already recommends including crypto agility as an evaluation criterion for project proposals. A citation for this claim would be helpful (the reference to NIST CSWP.39 is given, but the specific recommendation could be pinpointed).","section":"§1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 1 lists 'Basis' for each model. For FS-ISAC, the basis is 'CAMM [19]', but the text notes that FS-ISAC extends CAMM. For Deloitte, the basis is 'NIST CSF 2.0 [24]'. For ATIS, the basis is listed as '−' (none). It would be clearer to state 'None' or 'Independent' rather than a dash.","section":"§3.1, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper states that the three industry models do not document their design process or validation. While this is true, it may be worth noting that industry white papers often do not include such documentation, and this is not necessarily a flaw if the model is intended as a practical tool rather than a rigorous academic framework. The authors could acknowledge this context.","section":"§4.1, DP 1.1f","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper notes that ATIS does not clearly distinguish between KPIs and metrics. This is a valid observation, but the authors could provide a specific example of the ambiguity to illustrate the point.","section":"§4.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The recommendation 'Build on an established base, but carry its rigor through' is well-taken. However, the paper could also note that building on an established base may inherit its limitations (e.g., Deloitte inherits the CSF's domain neutrality, which may not be suitable for all contexts).","section":"§5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The guidance for selecting a model is useful, but it could be more concrete. For example, the paper could provide a decision tree or a table mapping organizational characteristics (e.g., sector, existing framework usage, technical expertise) to recommended models.","section":"§5.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The authors mention that 'the process of the evaluation with the Design Principles also led to discussions about the Design Principles itself.' This is an interesting point, but it is not elaborated. A brief comment on what aspects of the DPs might need refinement for crypto agility would strengthen the paper.","section":"§5.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Reference [2] is listed as 'Anonymous Authors, Digital Appendix...' with a DOI. This appears to be a placeholder for the supplemental material. The authors should ensure that the supplemental material is accessible and that the reference is updated if necessary.","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Leonie,\n\nThis paper does what it says on the tin: it takes the three industry frameworks that NIST cites as crypto agility maturity models (FS-ISAC, ATIS, Deloitte) and runs them through the Pöppelbuß & Röglinger design principles. The methodology is transparent, the per-principle verdicts are documented, and the authors are upfront about their own subjectivity. The main finding—that none of the three provides intersubjectively verifiable assessment criteria or integrated prescriptive guidance—is convincingly demonstrated for the lens they chose.\n\nCredit where it's due: the paper is well-structured, the limitations section is honest, and the comparative angle (including the earlier CAMM evaluation) gives practitioners a useful map. The recommendations for future model builders (build on a base but carry its rigor, provide verifiable criteria, integrate migration guidance) are sensible and grounded in the evidence.\n\nNow the soft spot, and it's the one the stress-test note flags. The paper evaluates FS-ISAC, ATIS, and Deloitte as maturity models subject to the full set of Pöppelbuß & Röglinger DPs. But the authors themselves state in §5 that 'none of the three models explicitly positions itself as a formal maturity model' and that ATIS 'would require significant additions or modifications to meet the strict definition of a maturity model.' The paper acknowledges this tension but then proceeds to evaluate them as if they were intended to be full maturity models. If FS-ISAC is a sector-specific extension, ATIS a KPI framework, and Deloitte a CSF community profile, then failing DPs 2.x and 3.x is not necessarily a design flaw—it's a mismatch between the evaluation lens and the artifact's purpose. The paper's strongest result (systematic failure on descriptive/prescriptive DPs) is therefore partly an artifact of the chosen benchmark, not a pure finding about the models' quality for their intended use.\n\nThat said, the paper does not hide this. It flags the issue in §5 and discusses it. The question is whether the central conclusion is load-bearing on this assumption. I think it is, but the authors are aware of it and the paper is still useful as a structured comparison. The reader's verdict (ACCEPT, high confidence) is reasonable if you accept the lens. I'd be more cautious: the paper is a solid evaluation of these artifacts against a particular framework, but the headline claim should be read with the lens-mismatch caveat firmly in mind.\n\nWho is this for? Practitioners choosing among the models, and researchers designing future crypto agility maturity models. It deserves peer review because the methodology is sound and the topic is timely, but a reviewer should push on whether the evaluation criteria are appropriate for artifacts that never claimed to be maturity models.\n\nRecommendation: Accept with minor revisions. The authors should strengthen the discussion of the lens-mismatch and consider whether the central claim needs qualification. The paper is honest enough that this is fixable.","headline":"Systematic evaluation of three industry crypto agility models against design principles; the central finding holds up, but the stress-test concern about applying a full maturity-model lens to artifacts that never claimed to be maturity models is real and acknowledged by the authors themselves.","tokens_in":17979,"tokens_out":733,"duration_ms":6741},"pith_extraction":{"msc":["68M25"],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"NIST-cited crypto agility models lack reproducible criteria","keywords":["crypto agility","maturity model","post-quantum cryptography","NIST","design principles","Pöppelbuß and Röglinger","FS-ISAC","ATIS"],"falsifier":"If a practitioner applied the ATIS model to the same organization twice and obtained the same maturity score both times without additional calibration, that would challenge the paper's claim that the models lack intersubjectively verifiable criteria. Conversely, if two independent assessors applying the same model to the same organization reach different conclusions, that would support the paper's central finding.","tokens_in":20206,"feed_emoji":"⚠️","tokens_out":1372,"duration_ms":18907,"temperature":0.7,"pith_summary":"The paper evaluates three industry-driven crypto agility maturity models referenced by NIST against established design principles for maturity models. It finds that while the models vary in meeting basic design principles according to their intended purposes, they systematically fail to provide intersubjectively verifiable assessment criteria and integrated prescriptive guidance for improving crypto agility. This matters because NIST's references give these models influence over organizational planning, governance, and procurement, yet without reproducible criteria and clear improvement paths they cannot support consistent assessments or guide cryptographic migrations.","feed_headline":"NIST-cited crypto agility models lack reproducible criteria","feed_subtitle":"Systematic evaluation finds three industry frameworks fail to provide verifiable assessment or integrated improvement guidance.","key_machinery":"The evaluation framework is the set of Design Principles for maturity models proposed by Pöppelbuß and Röglinger (2011), comprising basic principles (positioning, maturity constructs, domain definitions, documentation), descriptive principles (intersubjectively verifiable criteria and assessment methodology), and prescriptive principles (improvement measures, decision calculus, decision methodology). Two researchers applied these principles to each model through joint discussion, with tie-breaking input from a senior researcher, and the results were recorded in a per-principle verdict table.","core_discovery":"The three industry-driven crypto agility maturity models (FS-ISAC, ATIS, Deloitte) referenced in NIST's 2025 crypto agility white paper systematically fail to provide intersubjectively verifiable assessment criteria (descriptive design principles) and integrated guidance for improving crypto agility (prescriptive design principles), as evaluated against the established maturity model design principles of Pöppelbuß and Röglinger. The models diverge on basic design principles in ways that reflect their differing intended purposes and target audiences, but the descriptive and prescriptive shortcomings are consistent across all three models and also echo earlier findings for the academic CAMM. The paper concludes that without measurable, reproducible criteria and explicit decision logic linking maturity levels to improvement steps, these models can organize discussion but cannot support reproducible assessments or determine the next migration step.","pith_inferences":["The paper's finding that all three models fail prescriptive principles suggests that the concept of 'maturity' in crypto agility may be inherently multi-dimensional and resistant to the single linear progression assumed by traditional maturity models, a tension the authors acknowledge but do not resolve.","NIST's decision to reference these models as 'illustrative industry initiatives' without evaluating their design quality could inadvertently legitimize frameworks that lack the rigor needed for their intended governance role; the paper implies this gap needs attention.","The recurring measurement challenge identified—that many relevant properties of crypto agility are inherently difficult to operationalize—points to a need for fundamental research on measurable indicators of cryptographic agility before maturity models can achieve descriptive validity.","The paper's methodology, relying on researcher judgment rather than empirical application by target users, means its conclusions about practical applicability remain tentative; a follow-up study applying the models in real organizational settings could validate or challenge the design-principle-based findings."],"forward_implications":["Practitioners cannot rely on any of the evaluated models to produce reproducible maturity assessments, since even the most measurement-oriented model (ATIS) provides explicit scoring formulas for only 5 of its 23 metrics.","Organizations using these models for governance or procurement decisions risk inconsistent evaluations that depend on the subjective judgment of individual assessors.","Future crypto agility maturity model developers should build on established foundations while preserving rigor, provide verifiable criteria with explicit measurement inputs and transformation rules, integrate migration guidance into the maturity levels themselves, and serve both leadership and technical audiences with appropriate granularity.","The systematic failure across all models suggests that the linear maturity-stage paradigm may be ill-suited to the multi-dimensional nature of crypto agility, consistent with Rameshan and Messmer's argument for orthogonal dimensions at the application level.","If crypto agility maturity models are to inform regulatory guidance or large-scale PQC migration strategies, they must be redesigned to include measurable criteria and explicit decision logic that links assessment results to concrete improvement paths."],"supporting_citations":[{"why":"Supplies the Design Principles for maturity models that form the entire evaluation framework.","marker":"[26]"},{"why":"Provides the prior evaluation of the CAMM that the paper synthesizes with its own results and uses as a baseline comparison.","marker":"[36]"},{"why":"The NIST white paper whose references to the three evaluated models motivate the analysis.","marker":"[4]"},{"why":"The CAMM, which FS-ISAC extends and which provides the academic baseline for comparison.","marker":"[19]"},{"why":"The FS-ISAC model, one of the three evaluated models.","marker":"[16]"},{"why":"The ATIS model, one of the three evaluated models.","marker":"[3]"},{"why":"The Deloitte model, one of the three evaluated models.","marker":"[10]"},{"why":"Provides the strict definition of a maturity model used to note that ATIS may not qualify as one.","marker":"[5]"},{"why":"Further develops and applies the Design Principles, used to calibrate interpretations during evaluation.","marker":"[29]"},{"why":"Argues against a single linear hierarchy for crypto agility at the application level, cited to support the observation that the linear paradigm may be inappropriate.","marker":"[28]"}],"fun_headline_variants":["Crypto agility maturity models fail reproducibility test","NIST-cited models lack measurable maturity criteria","Design flaws found in crypto agility maturity models","Industry crypto models lack verifiable assessment","Crypto agility models show recurring design weaknesses"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The Design Principles of Pöppelbuß and Röglinger are the appropriate and sufficient framework for evaluating crypto agility maturity models, and the researchers' subjective application of these principles yields reliable conclusions about model quality.","fun_headline_variants_meta":{"raw":{"variants":["Crypto agility maturity models fail reproducibility test","NIST-cited models lack measurable maturity criteria","Design flaws found in crypto agility maturity models","Industry crypto models lack verifiable assessment","Crypto agility models show recurring design weaknesses"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":7.3e-05,"raw_usage":{"cost":7.33642e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":7.33642e-05,"upstream_inference_prompt_cost":6.31722e-05,"upstream_inference_completions_cost":1.0192e-05},"total_tokens":955,"prompt_tokens":899,"completion_tokens":56,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":899,"tokens_out":56,"cost_source":"openrouter_usage","duration_ms":2170,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T07:37:29.083588+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a practitioner applied the ATIS model to the same organization twice and obtained the same maturity score both times without additional calibration, that would challenge the paper's claim that the models lack intersubjectively verifiable criteria. Conversely, if two independent assessors applying the same model to the same organization reach different conclusions, that would support the paper's central finding.","supporting_citations":[{"cited_title":"In: ECIS Proceedings","cited_arxiv_id":null,"evidence_quote":"Supplies the Design Principles for maturity models that form the entire evaluation framework."},{"cited_title":"Quantum trajectories and measurements in continuous time: the diﬀusive case , vol. 782, Springer-Verlag Berlin Heidelberg, doi:10.1007/978-3- 642-01298-3","cited_arxiv_id":null,"evidence_quote":"Provides the prior evaluation of the CAMM that the paper synthesizes with its own results and uses as a baseline comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The NIST white paper whose references to the three evaluated models motivate the analysis."},{"cited_title":"In: Foundations and Practice of Security","cited_arxiv_id":null,"evidence_quote":"The CAMM, which FS-ISAC extends and which provides the academic baseline for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The FS-ISAC model, one of the three evaluated models."},{"cited_title":"org / resources / strategic - framework - for - crypto - agility-and-quantum-risk-assessment/","cited_arxiv_id":null,"evidence_quote":"The ATIS model, one of the three evaluated models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Deloitte model, one of the three evaluated models."},{"cited_title":"Business & information systems engineering1(3) (2009).https://doi.org/10.1007/s12599-009-0044-5 Maturity2: Assessing the Maturity of Crypto Agility Maturity Models 19","cited_arxiv_id":null,"evidence_quote":"Provides the strict definition of a maturity model used to note that ATIS may not qualify as one."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Further develops and applies the Design Principles, used to calibrate interpretations during evaluation."},{"cited_title":"Code generation vi a higher-order rewrite systems","cited_arxiv_id":null,"evidence_quote":"Argues against a single linear hierarchy for crypto agility at the application level, cited to support the observation that the linear paradigm may be inappropriate."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}