pith. sign in

arxiv: 2606.18405 · v1 · pith:GLDURG7Onew · submitted 2026-06-16 · 💻 cs.CR

Evaluating the Effectiveness of LLMs in Aiding Compliance Testing of PKCS#1-v1.5

Polina Kozyreva , Endadul Hoque This is my paper

Pith reviewed 2026-06-26 23:55 UTC · model grok-4.3

classification 💻 cs.CR
keywords PKCS#1 v1.5compliance testingLLM code synthesisspecification violationscryptographic librarieshallucinationTLV encodingsignature verification
0
0 comments X

The pith

LLM code synthesis for PKCS#1 v1.5 testing reproduces 10 of 13 known violation categories but is limited by hallucinations in 82.5% of scripts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines whether grammar-level mutations paired with LLM-generated test code can locate specification violations in implementations of the PKCS#1 v1.5 signature verification standard. It runs the method across 48 cryptographic libraries and measures results against an existing formal oracle. The evaluation reproduces nearly all previously known violation types, including every signature forgery category, and identifies one additional discrepancy. Hallucination in the LLM outputs turns out to be the dominant obstacle rather than the choice of mutation strategy. This matters because manual construction of compliance tests for binary protocols is costly, and the findings clarify when LLM assistance can be relied upon in security-critical testing.

Core claim

The paper establishes that grammar-level mutation combined with LLM-based code synthesis reproduces 10 of 13 non-trivial specification violation categories in PKCS#1 v1.5 signature verification, including all five signature forgery categories, and uncovers one previously unreported discrepancy across 48 library implementations. LLM hallucination occurs in 82.5% of the generated scripts and constitutes the primary limit on effectiveness. Structural mutations achieve 13.3% fidelity while constraint mutations reach 30.3% correctness yet suffer the highest rate of complete omissions at 8.1%. This produces a large gap between operational reliability of 99.8% and semantic fidelity of 17.5%.

What carries the argument

Grammar-level mutation combined with LLM-based code synthesis to generate test scripts targeting both structural and semantic constraints in TLV-encoded protocols.

If this is right

  • All five signature forgery categories can be surfaced by the hybrid mutation-plus-LLM method.
  • Hallucination rates differ systematically by mutation category, with structural changes showing lower fidelity than constraint changes.
  • High operational reliability does not translate to high semantic correctness, so generated scripts still require human review for critical uses.
  • The approach can locate discrepancies beyond those already known to the Morpheus oracle.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar LLM-assisted pipelines may work better on protocols whose constraints are mostly syntactic rather than deep semantic.
  • Detection of the five hallucination types could be automated to filter low-fidelity scripts before execution.
  • The observed fidelity gap suggests that purely LLM-driven test generation will remain supplementary rather than replacement for oracle-based methods in high-assurance settings.
  • Extending the same mutation categories to other ASN.1 or TLV protocols would test whether the hallucination patterns generalize.

Load-bearing premise

The Morpheus oracle supplies a complete and unbiased ground truth for every specification violation present in the selected libraries and mutation categories.

What would settle it

Independent manual review or formal re-analysis of the scripts that produced the single newly reported discrepancy, checking whether the claimed violation actually exists in the library.

Figures

Figures reproduced from arXiv: 2606.18405 by Endadul Hoque, Polina Kozyreva.

Figure 1
Figure 1. Figure 1: Pipeline architecture. The mutator produces [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
read the original abstract

Testing implementations of binary protocols for specification compliance requires inputs that satisfy both structural and semantic constraints. Purely random generation and primitive mutations are often insufficient for exploring semantically meaningful behaviors in protocols that rely on Type-Length-Value (TLV) encoding, yet domain-specific compliance testing tools require deep protocol expertise and significant manual effort to construct. This work investigates whether grammar-level mutation combined with LLM-based code synthesis can serve as a viable, more generalizable approach to specification compliance testing. We evaluate the approach on PKCS#1 v1.5 signature verification -- a widely deployed TLV-encoded standard with a formally verified testing oracle (Morpheus) -- across 48 cryptographic library implementations. We reproduced 10 of 13 non-trivial specification violation categories previously identified by Morpheus, including all 5 signature forgery categories, and discovered 1 previously unreported discrepancy. We found that LLM hallucination -- occurring in 82.5% of generated scripts -- is the primary factor limiting effectiveness, not the mutation strategies. We identify five distinct hallucination types and show that their distribution varies systematically across mutation categories: structural mutations are implemented with 13.3% fidelity while constraint mutations achieve 30.3% correctness but suffer the highest rate of mutations being fully ignored (8.1%). These findings reveal a striking gap between operational reliability (99.8%) and semantic fidelity (17.5%), providing actionable guidance on when LLM-based code synthesis can be trusted in specification-driven testing pipelines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper evaluates whether grammar-level mutation combined with LLM-based code synthesis can aid specification compliance testing for PKCS#1 v1.5 signature verification. Across 48 cryptographic libraries and using the formally verified Morpheus oracle, the authors report reproducing 10 of 13 non-trivial violation categories (including all 5 signature forgery categories), discovering one previously unreported discrepancy, and finding that LLM hallucination occurs in 82.5% of generated scripts and is the primary limiter. They further report fidelity rates of 13.3% for structural mutations versus 30.3% for constraint mutations, an 8.1% rate of fully ignored mutations, and a gap between 99.8% operational reliability and 17.5% semantic fidelity, along with five hallucination types whose distribution varies by mutation category.

Significance. If the central empirical results hold, the work supplies concrete, actionable data on the practical limits of current LLMs for synthesizing compliance test code in TLV-encoded protocols. Strengths include the use of a formally verified oracle (Morpheus) and evaluation across a sizable sample of 48 libraries; these choices allow direct comparison against an external ground truth and lend weight to the reported reproduction counts and hallucination statistics. The identification of systematic differences in fidelity and hallucination types across mutation categories could inform future hybrid testing pipelines.

major comments (3)
  1. [Abstract and §4] Abstract and §4 (Results): The headline claim that 10 of 13 categories were reproduced and that hallucination (82.5%) is the primary limiter rests on the assumption that Morpheus supplies a complete enumeration of all non-trivial specification violations. No argument, cross-check against an independent oracle, or coverage analysis is supplied to establish that the three unreproduced categories reflect LLM shortcomings rather than gaps in the oracle; this directly affects attribution of the central numbers.
  2. [§3] §3 (Evaluation Setup): The reported fidelity split (13.3% structural vs. 30.3% constraint) and ignored-mutation rate (8.1%) are presented as generalizable findings, yet the manuscript provides no justification or sampling argument that the chosen 48 libraries and the structural/constraint mutation taxonomy are representative of the broader space of PKCS#1 v1.5 implementations or mutation strategies. Without such grounding, the quantitative claims cannot be extrapolated beyond the specific experimental setup.
  3. [Abstract] Abstract: Reproduction counts and hallucination percentages are reported without accompanying dataset details, full prompt templates, selection criteria for the 48 libraries, or error-bar / variance information. This absence prevents assessment of whether post-hoc filtering or oracle assumptions influence the 10/13 reproduction rate and 82.5% figure, which are load-bearing for the effectiveness conclusion.
minor comments (2)
  1. [§4] Clarify the exact definition and measurement procedure for the five hallucination types and the fidelity metric (e.g., how “implemented with fidelity” is scored against the oracle).
  2. [§4] Add a table or figure that cross-tabulates hallucination type against mutation category to make the “systematic variation” claim easier to inspect.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed comments. We address each major point below, indicating where the manuscript will be revised to strengthen the claims.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Results): The headline claim that 10 of 13 categories were reproduced and that hallucination (82.5%) is the primary limiter rests on the assumption that Morpheus supplies a complete enumeration of all non-trivial specification violations. No argument, cross-check against an independent oracle, or coverage analysis is supplied to establish that the three unreproduced categories reflect LLM shortcomings rather than gaps in the oracle; this directly affects attribution of the central numbers.

    Authors: We acknowledge that the manuscript does not supply an explicit cross-check, independent oracle comparison, or coverage analysis beyond citing Morpheus's formal verification. This leaves open the possibility that the three unreproduced categories reflect limitations in the oracle rather than LLM performance. In the revision we will add a dedicated paragraph in §4 (and a brief note in the abstract) that states the completeness assumption explicitly, summarizes the formal verification basis for Morpheus, and lists the lack of independent validation as a limitation of the reported 10/13 figure. revision: partial

  2. Referee: [§3] §3 (Evaluation Setup): The reported fidelity split (13.3% structural vs. 30.3% constraint) and ignored-mutation rate (8.1%) are presented as generalizable findings, yet the manuscript provides no justification or sampling argument that the chosen 48 libraries and the structural/constraint mutation taxonomy are representative of the broader space of PKCS#1 v1.5 implementations or mutation strategies. Without such grounding, the quantitative claims cannot be extrapolated beyond the specific experimental setup.

    Authors: The 48 libraries were chosen for diversity among widely deployed open-source and commercial cryptographic implementations, and the mutation taxonomy follows directly from the violation categories enumerated by Morpheus. However, the manuscript indeed contains no formal sampling argument or representativeness claim. In the revised §3 we will add explicit selection criteria, a short discussion of scope, and a statement that the fidelity and ignored-mutation statistics are tied to this concrete set of libraries and the Morpheus-derived taxonomy rather than asserted as universally generalizable. revision: yes

  3. Referee: [Abstract] Abstract: Reproduction counts and hallucination percentages are reported without accompanying dataset details, full prompt templates, selection criteria for the 48 libraries, or error-bar / variance information. This absence prevents assessment of whether post-hoc filtering or oracle assumptions influence the 10/13 reproduction rate and 82.5% figure, which are load-bearing for the effectiveness conclusion.

    Authors: We agree that the abstract and main text would benefit from greater transparency. The evaluation-setup section already lists the 48 libraries and high-level methodology, but full prompt templates, exact dataset construction steps, and any variance statistics are not provided. In the revision we will (i) expand §3 with the requested selection criteria and dataset details, (ii) add an appendix containing the complete prompt templates, and (iii) report any available variance information or explicitly note that experiments were single-run and therefore lack error bars. These additions will allow readers to evaluate the influence of filtering or oracle assumptions on the headline numbers. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical reproduction rates derived from direct experimental comparison to external oracle

full rationale

The paper reports experimental results on LLM-generated test scripts for PKCS#1 v1.5 compliance across 48 libraries, measuring reproduction of 10/13 violation categories from the Morpheus oracle plus hallucination rates. No derivation chain, equations, fitted parameters renamed as predictions, or self-referential definitions appear in the abstract or described claims. The central statistics (82.5% hallucination, fidelity splits) are computed directly from the generated scripts and oracle outputs rather than reducing to any input by construction. Morpheus is invoked as an independent formally verified oracle; even if prior work by the authors, the evaluation does not rely on a uniqueness theorem or ansatz smuggled via self-citation to force the outcome. This is a standard empirical study whose validity rests on oracle completeness and sample representativeness, not on circular logic.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The central claim rests on the unexamined assumption that the chosen oracle and library set are representative.

pith-pipeline@v0.9.1-grok · 5803 in / 1112 out tokens · 27553 ms · 2026-06-26T23:55:30.446094+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 4 canonical work pages

  1. [1]

    Bachir Bendrissou, Cristian Cadar, and Alastair F Donaldson. 2025. Grammar mutation for testing input parsers. ACM Transactions on Software Engineering and Methodology 34, 4 (2025), 1–21

    2025

  2. [2]

    Daniel Bleichenbacher. 1998. Chosen ciphertext attacks against protocols based on the RSA encryption standard PKCS# 1. In Annual international cryptology conference. Springer, 1–12

    1998

  3. [3]

    Muhammad Daniyal Pirwani Dar, Rob Lorch, Aliakbar Sadeghi, Vincenzo Sorcigli, Héloïse Gollier, Cesare Tinelli, Mathy Vanhoef, and Omar Chowdhury. 2025. Saecred: A state-aware, over-the-air protocol testing approach for discovering parsing bugs in SAE handshake implementations of COTS Wi-Fi access points. In 2025 IEEE symposium on security and privacy (SP)...

    2025

  4. [4]

    Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. 2020. {AFL++}: Combining incremental steps of fuzzing research. In 14th USENIX workshop on offensive technologies (WOOT 20)

    2020

  5. [5]

    S Mahmudul Hasan, Polina Kozyreva, and Endadul Hoque. 2024. Fuzze- val: Assessing fuzzers on generating context-sensitive inputs. arXiv preprint arXiv:2409.12331 (2024). Evaluating the Effectiveness of LLMs in Aiding Compliance Testing of PKCS#1-v1.5

    work page arXiv 2024

  6. [6]

    International Telecommunication Union. 2021. Information technology – ASN.1 encoding rules: Specification of Basic Encoding Rules (BER), Canonical Encoding Rules (CER) and Distinguished Encoding Rules (DER) . Recommendation X.690. ITU-T. https://itu.int

    2021

  7. [7]

    B Kaliski, J Jonsson, and A Rusch. 2016. RFC 8017: PKCS# 1: RSA Cryptography Specifications Version 2.2

    2016

  8. [8]

    Fang Liu, Yang Liu, Lin Shi, Zhen Yang, Li Zhang, Xiaoli Lian, Zhongqi Li, and Yuchi Ma. 2026. Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code. IEEE Transactions on Software Engineering (2026)

    2026

  9. [9]

    Kuangxiangzi Liu, Dhiman Chakraborty, Alexander Liggesmeyer, and Andreas Zeller. 2025. Synthesizing Precise Protocol Specs from Natural Language for Effective Test Generation. arXiv preprint arXiv:2511.17977 (2025)

    work page arXiv 2025

  10. [10]

    Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. Large Language Model guided Protocol Fuzzing.. In NDSS

    2024

  11. [11]

    Peach Tech. 2008. Peach Fuzzer. https://peachtech.gitlab.io/peach-fuzzer- community/

    2008

  12. [12]

    Joshua Pereyda. 2014. boofuzz: Network Protocol Fuzzing for Humans. https: //github.com/jtpereyda/boofuzz

    2014

  13. [13]

    Van-Thuan Pham, Marcel Böhme, and Abhik Roychoudhury. 2020. Aflnet: A greybox fuzzer for network protocols. In 2020 IEEE 13th international conference on software testing, validation and verification (ICST) . IEEE, 460–465

    2020

  14. [14]

    Yujin Potter, Wenbo Guo, Zhun Wang, Tianneng Shi, Hongwei Li, Andy Zhang, Patrick Gage Kelley, Kurt Thomas, and Dawn Song. 2025. Frontier AI’s Impact on the Cybersecurity Landscape. arXiv preprint arXiv:2504.05408 (2025)

    work page arXiv 2025

  15. [15]

    Wenxuan Shi, Yunhang Zhang, Xinyu Xing, and Jun Xu. 2024. Harnessing large language models for seed generation in greybox fuzzing. arXiv preprint arXiv:2411.18143 (2024)

    work page arXiv 2024

  16. [16]

    Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4all: Universal fuzzing with large language models. In Pro- ceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

    2024

  17. [17]

    Moosa Yahyazadeh, Sze Yiu Chau, Li Li, Man Hong Hue, Joyanta Debnath, Sheung Chiu Ip, Chun Ngai Li, Endadul Hoque, and Omar Chowdhury. 2021. Morpheus: Bringing the (pkcs) one to meet the oracle. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security . 2474–2496

    2021

  18. [18]

    José Antonio Zamudio Amaya, Marius Smytzek, and Andreas Zeller. 2025. FAN- DANGO: evolving language-based testing. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 894–916

    2025

  19. [19]

    __main__

    Ziyao Zhang, Chong Wang, Yanlin Wang, Ensheng Shi, Yuchi Ma, Wanjun Zhong, Jiachi Chen, Mingzhi Mao, and Zibin Zheng. 2025. Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 481–503. Polina Kozyreva and Endadul Hoque Appendix A Prompt Templates This append...

    2025