REVIEW 4 major objections 4 minor 31 references
Hidden Ciphers and Where to Find Them: Static Discovery and Assessment of Cryptographic Assets in Software
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Rule-based static analysis grounded in a three-way asset taxonomy can provide practical cryptographic transparency for deployed software systems.
desk verdict Useful taxonomy and scanner-independent rule repository for crypto-asset discovery, with an evaluation that is honest but weaker than the headline F1 suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the three-way discovery classification (Crypto-Material, Crypto-Artifacts, Crypto-Invocations) paired with the two-way assessment classification (Weakness, Vulnerability). Its load-bearing role is to make rule design uniform: every discovery rule in Crypistry maps to exactly one asset category, and every assessment rule operates on normalized CryptoComponent objects rather than raw files. This uniformity, plus the separation of the rule repository from the scanner pipeline, is what lets Crypsy recognize new file formats and library APIs by adding rules instead of changing scanner code, and it is what allows the same classification to drive both occurrence-level discovery and CWE/CVE-based assessment.
What would settle it
Run Crypsy on a set of Java, C/C++, or Python services whose cryptographic assets have been enumerated by an independent, exhaustive ground-truth process; if the scanner's recall falls well below the 0.84–0.96 range reported on the synthetic benchmark, or if it misses large classes of artifacts or material present in those ecosystems, the paper's transparency claim would be refuted for those ecosystems.
Extended reading notes
Core claim
The central claim, as stated in Section 9, is that rule-based static discovery grounded in a structured classification can provide practical cryptographic transparency for deployed software systems. The classification defines three discovery categories—value-centric Crypto-Material (private keys, tokens, seeds, nonces), structure-centric Crypto-Artifacts (X.509 certificates, keystores, signed manifests), and context-centric Crypto-Invocations (source-level API calls, TLS/SSH configuration directives, dependency declarations)—plus two assessment categories, weakness (CWE-mapped) and vulnerability (CVE-mapped). From this classification the paper derives Crypistry, a scanner-independent repository of 148 discovery and 66 assessment rules, and Crypsy, a four-phase scanner (rule loading, discovery, assessment, CBOM export) that produces CycloneDX v1.7 CBOM output. The supporting evidence is an occurrence-level F1 of 0.75 on the Cryben synthetic benchmark (0.86 when by-design counting differences are excluded, 0.95 at component granularity), 91% assessment recall, and a real-world scan of ten services that discovered 370 components, including six CVE-linked vulnerabilities and 52 post-quantum migration candidates. The paper is careful to scope the real-world recall as agreement with a manually compiled partial reference list rather than exhaustive coverage.
Load-bearing premise
The real-world evaluation supports the transparency claim only if the manually compiled partial reference list is representative of the cryptography actually deployed in the ten services; the paper itself reads its 100% partial recall as 'no manually identified asset was missed,' not complete coverage, and the claim is explicitly scoped to Ruby, Go, and the covered configuration formats.
Editorial extensions
If this is right
- Organizations could generate a CycloneDX CBOM automatically from source, configuration, and dependency files, replacing ad hoc manual surveys with a repeatable, evidence-linked inventory.
- Post-quantum migration planning can begin from a concrete candidate list (the 52 RSA/ECC keys, certificates, and TLS configurations found in the evaluation) instead of a blank sheet.
- Because detection and assessment knowledge live in a versioned rule repository, adding a new library, language, or CVE/CWE entry can be done without rewriting scanner pipeline logic.
- Weakness and vulnerability annotations arrive pre-mapped to CWE and CVE identifiers, so findings can plug directly into governance and vulnerability-management workflows.
Reading between the lines
- The classification itself is language-agnostic, so the same rule-repository procedure could be applied to Java, C/C++, and Python; the empirical claim, however, would need re-validation in those ecosystems before 'practical transparency' can be generalized beyond Ruby and Go.
- Because 43 of the 67 synthetic-benchmark false negatives are by-design counting differences, a component-level or CBOM-level scoring convention may be the fairer metric for inventory tools than occurrence-level F1.
- The roughly 0.3 actionable-weakness precision suggests a natural extension: add lightweight taint or dataflow tracking so weakness annotations like weak-hash or weak-PRNG only fire when the result is actually used in a security-sensitive context.
- A stronger test than the manual partial reference list would be to compare Crypsy's CBOM against an inventory built by multiple independent analyzers, including binary-level and dynamic tools, to measure true coverage in the presence of unparseable or unsupported files.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a classification-driven static approach for discovering cryptographic assets in software. It introduces three discovery categories (Crypto-Material, Crypto-Artifacts, Crypto-Invocations) and two assessment categories (Weakness, Vulnerability), derives a scanner-independent rule repository called Crypistry, implements a static scanner called Crypsy, and evaluates the artifacts on a synthetic benchmark (Cryben) and on ten deployed services. The reported headline results are an F1 of 0.75 for benchmark discovery (0.86 excluding by-design counting differences, 0.95 at component level), 91% assessment recall on expected identifiers, and, in the real-world scan, 370 discovered cryptographic components with six CVE-linked vulnerabilities and 52 post-quantum migration candidates. The paper is transparent that the real-world ground truth is partial and that language coverage is currently limited to Ruby, Go, and selected configuration formats.
Significance. The work addresses a real and timely gap: organizations lack structured inventories of cryptographic assets for governance and post-quantum migration. The main strengths are the clear separation between classification, rule repository, scanner, and benchmark; the explicit separation of by-design counting differences from true detection gaps in the synthetic evaluation; the head-to-head comparison with CBOMkit-hyperion; and the candid discussion of limitations in Sections 7.2 and 8. If the central claim is read as scoped to the supported ecosystems, the paper provides a useful, reproducible baseline for CBOM-oriented static crypto discovery. The open supplementary material strengthens the contribution by making the artifacts independently inspectable. The main weaknesses are that the real-world evaluation cannot bound missed assets because of its partial, author-built ground truth, and that the synthetic benchmark's ground truth is not fully independent of the taxonomy that generated the rules, so the reported correctness numbers partly measure internal consistency.
major comments (4)
- [§9 and Abstract] The central claim that 'rule-based static discovery, grounded in a structured classification, can provide practical cryptographic transparency for deployed software systems' is stated at a level of generality that the evidence does not support. Section 8 explicitly limits the empirical evidence to Ruby, Go, and selected configuration formats and states that Java, C/C++, and Python are 'not yet demonstrated.' Since those ecosystems hold a substantial fraction of enterprise cryptography, the claim should either be explicitly scoped to the evaluated ecosystems or be accompanied by evidence of transferability. As written, the abstract and conclusion assert practical transparency for deployed systems broadly, while the only deployed evidence comes from ten services dominated by Ruby, Go, and configuration files.
- [§7.2, real-world recall] Section 7.2 reports a 'partial recall of 100%' against a manually compiled reference list and correctly states that this is not exhaustive. However, this design cannot bound the number of missed assets; it only establishes that no asset already known to the authors was missed. The subsequently emphasized counts (370 components, 52 post-quantum candidates, six CVE-linked vulnerabilities) are therefore lower bounds, not demonstrated inventories. Because the manual inspection was conducted by the authors on the same repositories that were scanned, and because unsupported languages and unparsed file types are excluded from the scan, the real-world evaluation does not by itself justify the 'practical cryptographic transparency' conclusion. The manuscript should either report the fraction and language distribution of parsed versus unparsed files and give a best-effort estimate of crypto-bearing content in unparsed files, or explicitly weaken the claim to transparency for the supported languages and file types.
- [§7.1, benchmark construction] Section 7.1 states that Cryben's ground truth is a CycloneDX CBOM 'constructed independently of Crypsy,' but it does not state that the ground truth was constructed independently of Crypistry or of the classification in Section 4. Given the paper's own account in Section 3 that the classification and the rule repository co-evolved, the benchmark annotations and the discovery rules plausibly share the same conceptual source. The reported F1 therefore partly measures the scanner's consistency with the authors' own taxonomy rather than agreement with an external standard. This is a threat to the 'correctness' interpretation of the benchmark numbers. The authors should either base the benchmark ground truth on an independent source (for example, a second annotation team or a pre-existing inventory constructed without reference to Crypistry), or explicitly frame the benchmark results as a consistency measure and discuss how much of the reported performance is attributable to the shared taxonomy.
- [§7.2, actionable weakness precision] The abstract highlights that the scanner 'correctly annotates 91% of expected weaknesses and vulnerabilities,' but that figure is benchmark-level assessment recall. In the real-world evaluation, Section 7.2 reports that of the 91 components receiving a CWE-mapped weakness annotation, only about 0.3 are actionable after manual review, because weak-hash and weak-PRNG rules fire on non-security digests and identifiers. This discrepancy is material to the assessment contribution. The manuscript should present the real-world actionable-weakness precision prominently and clarify in the abstract or conclusion that the 91% figure is a recall measure against a known-identifier benchmark, not a precision or actionability measure on real-world findings. Without this clarification, the assessment claim reads as stronger than the evidence supports.
minor comments (4)
- [Table 4] As typeset, the Material and Artifacts rows appear to have ten values for eleven columns, so the per-service counts for Redis and PostgreSQL are not fully recoverable from the table. Please realign the columns and recheck the row-wise sums against the P column.
- [Figure 5] The CVE column labels are abbreviated to year ranges (CVE-2011, CVE-2013, CVE-2015, CVE-2019). Consider listing the full CVE identifiers or providing a legend so that the assessment coverage is directly interpretable.
- [Table 3] The summary rows use the labels 'P Findings', 'P excl. by-design', and 'P component level', but 'P' is elsewhere used for precision. Rename these rows (for example, 'Overall', 'Overall excl. by-design', 'Component level') to avoid confusion with the precision column.
- [§5.2] The sentence 'A single rule is not bound to a single file' is unclear on first reading; consider rephrasing to explain that a rule can match multiple files and multiple occurrences, which is what makes the rule repository compact.
Circularity Check
No significant circularity: the scanner is evaluated against an explicitly partial manual ground truth and a synthetic benchmark constructed independently of the scanner implementation; the paper discloses its limitations and does not present fitted parameters or self-citation chains as evidence.
full rationale
The paper's derivation chain is classification -> rule repository -> scanner -> evaluation. This is an implementation and validation pipeline, not a derivation whose conclusion is equivalent to its inputs. The synthetic benchmark Cryben is stated to have "a CycloneDX v1.7 CBOM constructed independently of Crypsy" (Section 7.1), so the scanner is not being scored against its own output. The reported F1 of 0.75, with 67 false negatives and 20 false positives, shows that the evaluation is not forced by construction; the paper even decomposes the results into by-design counting differences and genuine detection gaps. The real-world evaluation explicitly states that the 100% recall "reflects agreement with our manually built partial ground truth, not true recall against an exhaustive inventory" (Section 7.2), and the abstract and conclusion repeat this caveat. Thus the central transparency claim is scoped to the evaluated languages and to partial ground truth, which is a validity limitation rather than circular reasoning. The paper's self-citations (e.g., [12], [15], [19], [22]) are used for motivation and framing, not as load-bearing evidence for the scanner's correctness. No uniqueness theorems are imported from the authors' prior work, and no ansatz is smuggled in via citation. The benchmark ground truth does share the paper's taxonomy, which limits how much the F1 score validates the classification itself, but the paper does not claim that the benchmark validates the classification; it claims the scanner correctly implements the classification on a constructed corpus and on deployed services, and the disclosed false negatives and language-scope limitations make that claim non-circular.
Assumptions & free parameters
assumptions (4)
- domain assumption Static at-rest analysis can discover the practically relevant cryptographic assets of deployed software systems.
- ad hoc to paper The three-category classification (Material, Artifact, Invocation) is complete for static crypto asset discovery.
- domain assumption The synthetic benchmark Cryben's ground truth is a valid proxy for real-world cryptographic assets.
- domain assumption CWE and CVE mappings can be applied to statically detected findings without usage-context analysis.
Cite this review
Pith. "Pith review of Hidden Ciphers and Where to Find Them: Static Discovery and Assessment of Cryptographic Assets in Software." pith.science (2026). https://pith.science/paper/MUYHK2W7
@misc{pith2026260804857,
author = {Pith},
title = {Pith review of: Hidden Ciphers and Where to Find Them: Static Discovery and Assessment of Cryptographic Assets in Software},
year = {2026},
howpublished = {\url{https://pith.science/paper/MUYHK2W7}},
note = {Machine review of arXiv:2608.04857}
}
read the original abstract
Modern software systems rely on cryptography for data protection, authentication, and trust establishment, yet organizations often lack a structured view of the cryptography deployed across source code, configuration, dependencies, and cryptographic files. This lack of visibility complicates security governance and post-quantum migration planning. This paper presents a static approach for discovering and assessing cryptographic assets in software systems. We introduce a classification of Crypto-Material, Crypto-Artifacts, and Crypto-Invocations, derive an extensible scanner-independent rule repository from it, and implement a static scanner that applies these rules to produce CBOM-oriented output. We evaluate the approach on a synthetic benchmark with known ground truth and on a real-world infrastructure of ten deployed services. The scanner achieves an F1 score of 0.75 for asset discovery and correctly annotates 91% of expected weaknesses and vulnerabilities. In the realworld setting, it processes 57 610 files in under six minutes and discovers 370 cryptographic assets, including six CVE-linked vulnerabilities and 52 post-quantum migration candidates. Real-world coverage is assessed against a manually compiled reference list rather than an exhaustive one. These results show that classification-driven static discovery can provide practical cryptographic transparency for governance and post-quantum migration planning.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Supplementary materials, https://osf.io/4bc5p/overview?view_only= dc5aab1d48d9466f8260fe4e2a8e0e79
- [2]
-
[3]
Cybersecurity White Paper CSWP 15, NIST (Apr 2021)
Barker, W.C., Polk, W.T., Souppaya, M.: Getting ready for post-quantum cryp- tography: Exploring challenges associated with adopting and using post-quantum cryptographic algorithms. Cybersecurity White Paper CSWP 15, NIST (Apr 2021). https://doi.org/10.6028/NIST.CSWP.15
-
[4]
CBOMkit Contributors: CBOMkit.https://github.com/cbomkit/cbomkit (2025)
work page 2025
-
[5]
In: IEEE/ACM International Conference on Automated Software Engineering (ASE 2025)
Cho, S., Yu, S., Woo, S.: Cryptbara: Dependency-guided detection of Python cryptographic API misuses. In: IEEE/ACM International Conference on Automated Software Engineering (ASE 2025). pp. 1578–1590 (2025).https://doi.org/10. 1109/ASE63991.2025.00133
arXiv 2025
-
[6]
In: 2021 8th NAFOSTED Conference on Information and Computer Science (NICS)
Choudhari, A., Guilley, S., Karray, K.: CRYScanner: Finding cryptographic libraries misuse. In: 2021 8th NAFOSTED Conference on Information and Computer Science (NICS). pp. 230–235 (2021).https://doi.org/10.1109/NICS54270.2021.9701469
arXiv 2021
-
[7]
CISA, NSA, NIST: Quantum-readiness: Migration to post-quantum cryp- tography. https://media.defense.gov/2023/Aug/21/2003284212/-1/-1/0/ CSI-QUANTUM-READINESS.PDF(2023), cybersecurity Information Sheet
work page 2023
-
[8]
In: 2024 Annual Computer Security Applica- tions Conference
Fan, Y., Biswas, P., Garman, C.: R+r: A systematic study of cryptographic function identification approaches in binaries. In: 2024 Annual Computer Security Applica- tions Conference. pp. 1092–1108 (2024).https://doi.org/10.1109/ACSAC63791. 2024.00089
Show all 31 references
-
[9]
IEEE Transactions on Software En- gineering50(5), 1118–1129 (2024)
Frantz, M., Xiao, Y., Pias, T.S., Meng, N., Yao, D.: Methods and benchmark for detecting cryptographic API misuses in Python. IEEE Transactions on Software En- gineering50(5), 1118–1129 (2024). https://doi.org/10.1109/TSE.2024.3377182
2024
-
[10]
Scandinavian Journal of Information Systems19(2), 87–92 (2007)
Hevner, A.R.: A three cycle view of design science research. Scandinavian Journal of Information Systems19(2), 87–92 (2007)
2007
-
[11]
In: MIS Quarterly
Hevner, A.R., March, S.T., Park, J., Ram, S.: Design Science in Information Systems Research. In: MIS Quarterly. vol. 28, pp. 75–105 (2004)
2004
-
[12]
In: Migration and Agility in Cryp- tographic Systems (MAgiCS 2026)
Hirsch, E., Raab, K.: Architecture-derived CBOMs for cryptographic migration: A security-aware architecture tradeoff method. In: Migration and Agility in Cryp- tographic Systems (MAgiCS 2026). CCIS, vol. 3002, pp. 20–39. Springer (2026). https://doi.org/10.1007/978-3-032-28946-9_2
2026 doi
-
[13]
IEEE Transactions on Software Engineering47(11), 2382–2400 (2021)
Krüger, S., Späth, J., Ali, K., Bodden, E., Mezini, M.: Crysl: An extensible approach to validating the correct usage of cryptographic APIs. IEEE Transactions on Software Engineering47(11), 2382–2400 (2021). https://doi.org/10.1109/TSE. 2019.2948910
2021
-
[14]
Master’s thesis, University of Jyväskylä (2024),https://urn
Leirimaa, K.: Supporting PQC migration and cryptographic agility with automated CBOM generation. Master’s thesis, University of Jyväskylä (2024),https://urn. fi/URN:NBN:fi:jyu-202411016889
2024
-
[15]
arXiv preprint arXiv:2408.05997 (2024).https://doi.org/10.48550/arXiv.2408.05997
Loebenberger,D.,Gazdag,S.L.,Herzinger,D.,Hirsch,E.,Näther,C.,Steghöfer,J.P.: On the formalization of cryptographic migration. arXiv preprint arXiv:2408.05997 (2024).https://doi.org/10.48550/arXiv.2408.05997
-
[16]
In: 30th USENIX Security Symposium (USENIX Security 21)
Meijer, C., Moonsamy, V., Wetzels, J.: Where’s crypto?: Automated identification and classification of proprietary cryptographic primitives in binary code. In: 30th USENIX Security Symposium (USENIX Security 21). pp. 555–572 (2021) 20 C. Näther, E. Hirsch
2021
- [17]
-
[18]
ACM Comput
Mousavi, Z., Islam, C., Babar, M.A., Abuadbba, A., Moore, K.: Detecting misuse of security APIs: A systematic review. ACM Comput. Surv.57(12) (2025).https: //doi.org/10.1145/3735968
2025 doi
-
[19]
arXiv preprint arXiv:2411.08781 (2024).https://doi.org/10.48550/arXiv.2411.08781
Näther,C.,Herzinger,D.,Steghöfer,J.P.,Gazdag,S.L.,Hirsch,E.,Loebenberger,D.: Toward a common understanding of cryptographic agility–a systematic review. arXiv preprint arXiv:2411.08781 (2024).https://doi.org/10.48550/arXiv.2411.08781
-
[20]
https://csrc.nist.gov/projects/ cryptographic-standards-and-guidelines(2016), accessed: 2026-05-20
National Institute of Standards and Technology: Cryptographic standards and guidelines. https://csrc.nist.gov/projects/ cryptographic-standards-and-guidelines(2016), accessed: 2026-05-20
2016
-
[21]
https://media.defense.gov/2025/May/30/2003728741/-1/-1/0/CSA_ CNSA_2.0_ALGORITHMS.PDF(Sep 2022), pP-22-1338, Version 1.0
National Security Agency: Announcing the commercial national security algorithm suite 2.0. https://media.defense.gov/2025/May/30/2003728741/-1/-1/0/CSA_ CNSA_2.0_ALGORITHMS.PDF(Sep 2022), pP-22-1338, Version 1.0
2025
-
[22]
IEEE Access12, 132107–132126 (2024).https://doi.org/10
Näther, C., Herzinger, D., Gazdag, S.L., Steghöfer, J.P., Daum, S., Loebenberger, D.: Migrating software systems toward post-quantum cryptography-a systematic literature review. IEEE Access12, 132107–132126 (2024).https://doi.org/10. 1109/ACCESS.2024.3450306
2024
-
[23]
In: 2021 IEEE Symposium on Security and Privacy (SP)
Piccolboni, L., Guglielmo, G.D., Carloni, L.P., Sethumadhavan, S.: CRYLOGGER: Detecting crypto misuses dynamically. In: 2021 IEEE Symposium on Security and Privacy (SP). pp. 1972–1989 (2021). https://doi.org/10.1109/SP40001.2021. 00010
2021
-
[24]
In: ACM SIGSAC Conference on Computer and Communications Security (CCS 2019)
Rahaman,S.,Yao,D.,Shahriar,A.B.M.,DeRuiter,A.,Chatterjee,S.:CryptoGuard: High precision detection of cryptographic vulnerabilities in massive-sized Java projects. In: ACM SIGSAC Conference on Computer and Communications Security (CCS 2019). pp. 2455–2472 (2019).https://doi.org...
2019
-
[25]
IEEE Access13, 4368–4380 (2025).https://doi.org/10.1109/ACCESS.2024.3524309
Rattanavipanon, N., Suaboot, J., Werapun, W.: A toolchain for assisting migration of software executables towards post-quantum cryptography. IEEE Access13, 4368–4380 (2025).https://doi.org/10.1109/ACCESS.2024.3524309
2025
-
[26]
org/abs/2204.06447
Schlichtig, M., Wickert, A.K., Krüger, S., Bodden, E., Mezini, M.: Cambench: Cryptographic API misuse detection tool benchmark suite (2022),https://arxiv. org/abs/2204.06447
2022 arXiv
-
[27]
In: Sicherheit 2024
Schmitt, N., Henrich, J., Heinz, D., Alnahawi, N., Wiesmaier, A.: On criteria and tooling for cryptographic inventories. In: Sicherheit 2024. vol. P-345, pp. 49–63 (2024).https://doi.org/10.18420/sicherheit2024_003
2024 doi
-
[28]
Proceedings of the 35th Annual Symposium on Foundations of Computer Science pp
Shor, P.W.: Algorithms for quantum computation: Discrete logarithms and factoring. Proceedings of the 35th Annual Symposium on Foundations of Computer Science pp. 124–134 (1994).https://doi.org/10.1109/SFCS.1994.365700
1994
-
[29]
In: Research in Attacks, Intrusions and Defenses (RAID 2024)
Wang, J., Guo, S., Diao, W., Liu, Y., Duan, H., Liu, Y., Liang, Z.: CrypTody: Cryptographic misuse analysis of IoT firmware via data-flow reasoning. In: Research in Attacks, Intrusions and Defenses (RAID 2024). pp. 579–593 (2024).https: //doi.org/10.1145/3678890.3678914
2024
-
[30]
Wind River Systems: Crypto detector (2020),https://github.com/Wind-River/ crypto-detector
2020
-
[31]
aes.NewCipher(key)
Zhang, Y., Kabir, M.M.A., Xiao, Y., Yao, D., Meng, N.: Automatic detection of Java cryptographic API misuses: Are we there yet? IEEE Transactions on Software Engineering49(1), 288–303 (2023). https://doi.org/10.1109/TSE.2022.3150302 Hidden Ciphers and Where to Find Them 21 A A...
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.