Pith. sign in

REVIEW 2 minor 32 references

NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction

T0 review · 0 major / 2 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NICE uses declarative Nix configurations to build vulnerable environments and supply machine-checkable proofs of exploitation.

desk verdict NICE gives a working Nix-based way to declaratively reproduce and machine-verify 19 CVEs, which is useful for reproducibility but rests on case-study evidence rather than deep metrics on assertion coverage. read the letter →

arxiv 2606.00625 v1 pith:MJBUGW7F submitted 2026-05-30 cs.CR

Minh-Luân Nguyen , Olivier Levillain , Julien Malka , Stefano Zacchiroli , Théo Zimmermann This is my paper · ORCID
classification cs.CR
keywords vulnerabilityreproductionNixCVEdeclarativeconfigurationmachine-checkableassertionsreproducibilitysecuritytesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents NICE, a framework that turns vulnerability reproduction into declarative recipes consisting of NixOS virtual-machine configurations, exploitation scripts, and assertions that confirm successful exploitation. These recipes are evaluated on 19 real-world CVEs covering different categories, attack vectors, and target types. A sympathetic reader would care because recreating the exact conditions needed to trigger a vulnerability becomes increasingly difficult over time, and the approach makes such reproductions shareable, reviewable, and durable for researchers, maintainers, and educators.

What carries the argument

Declarative NixOS virtual machine configurations combined with scripted exploitation scenarios and machine-checkable assertions that confirm exploitation.

What would settle it

A CVE for which a NICE recipe builds and runs but the assertions never pass, even after the Nix configuration is adjusted to match every detail reported in the original vulnerability description.

Watch

Extended reading notes

Core claim

NICE allows the production of concise recipes and integration tests that reproduce vulnerable environments and provide proofs of exploitation. A reproduced CVE comprises one or more NixOS virtual machine configurations, a scripted exploitation scenario, and machine-checkable assertions that provide factual evidence of exploitation. This design facilitates sharing, validation, review, and long-term reproducibility.

Load-bearing premise

Declarative Nix configurations can faithfully capture the runtime conditions needed for exploitation across diverse vulnerability classes without requiring substantial manual intervention or losing fidelity.

Editorial extensions

If this is right

  • Facilitates sharing, validation, review, and long-term reproducibility of vulnerability reproductions.
  • Applicable to security education and training such as creating cyber ranges.
  • Makes vulnerability reports easier to audit and verify.
  • Supports reproduction for user-space, system software, kernel, and graphical applications.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same declarative style could be ported to other configuration systems if Nix adoption remains limited in some security communities.
  • A public archive of verified NICE recipes could serve as a living test suite for vulnerability detection tools.
  • Integration with existing bug trackers might allow automatic generation of draft reproduction recipes from submitted reports.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The paper introduces NICE, a framework for reproducing software vulnerabilities using declarative NixOS virtual machine configurations, scripted exploitation scenarios, and machine-checkable assertions. It evaluates the approach on 19 diverse real-world CVEs spanning multiple CWE categories, attack vectors, and target types (user-space, kernel, graphical), claiming that the framework produces concise recipes and integration tests that reproduce vulnerable environments and provide proofs of exploitation, with applications to security education, training, and vulnerability reporting.

Significance. If the results hold, NICE offers a practical contribution to vulnerability reproduction by emphasizing declarative specifications and machine-checkable evidence, which could improve shareability, auditability, and long-term reproducibility. The evaluation across 19 CVEs with claims of working proofs is a strength, as is the focus on empirical coverage rather than untested assumptions. This aligns with needs in education (e.g., cyber ranges) and reporting.

minor comments (2)
  1. [Abstract] Abstract: The claim that NICE 'allows to produce concise recipes and integration tests' is central but lacks any quantitative metric (e.g., lines of code, comparison to manual setups) or definition of conciseness, weakening the reader's ability to assess the result.
  2. [Abstract] Abstract: Machine-checkable assertions are presented as providing 'factual evidence of exploitation' and 'proofs,' yet the abstract supplies no information on their implementation, what they assert, or observed failure modes; this detail is load-bearing for the reproducibility claim even if present in later sections.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive assessment of the work and recommendation for minor revision. The report provides no major comments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity identified

full rationale

The paper describes a declarative framework (NICE) for reproducing CVEs via NixOS VM configurations, scripted scenarios, and machine-checkable assertions, then evaluates it empirically on 19 real-world CVEs across multiple categories. No derivations, equations, fitted parameters, predictions, or self-citation chains appear in the load-bearing claims; the central results rest on direct case-study coverage rather than any reduction to inputs by construction. The work is therefore self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The paper introduces an engineering framework rather than a theoretical derivation; no free parameters, domain axioms, or invented entities are required or stated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction." pith.science (2026). https://pith.science/paper/MJBUGW7F

@misc{pith2026260600625,
  author       = {Pith},
  title        = {Pith review of: NICE: A Framework for Declarative and Machine-Checkable Vulnerability Reproduction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MJBUGW7F}},
  note         = {Machine review of arXiv:2606.00625}
}
read the original abstract

Reproducing software vulnerabilities is fundamental to security researchers, open-source maintainers, and educators. Yet, vulnerabilities remain hard to reproduce today, and even when they can be reproduced, recreating a software environment where the vulnerability can be exploited becomes harder and harder over time. We present NICE, the NIx CvE reproduction framework, which uses declarative recipes to build and automatically validate vulnerable environments. In NICE, a reproduced CVE comprises one or more NixOS virtual machine configurations, a scripted exploitation scenario, and machine-checkable assertions that provide factual evidence of exploitation. This design facilitates sharing, validation, review, and long-term reproducibility. We evaluate NICE on 19 diverse real-world CVEs spanning multiple CWE categories, attack vectors, and target types (user-space, system software, kernel, and graphical applications). We show that NICE allows to produce concise recipes and integration tests that reproduce vulnerable environments and provide proofs of exploitation. NICE is applicable to security education and training (e.g., creating cyber ranges), but also to vulnerability reporting, where its reproducibility and reviewability properties can make reports easier to audit and verify.

Figures

Figures reproduced from arXiv: 2606.00625 by the authors.

Figure 1
Figure 1. Example of a NixOS configuration. package versions will already be available from Nixpkgs. If not, it is easy to provide the missing package recipes. Furthermore, Nix supports a pinning mechanism that en￾sures that software environments will stay the same when re-deployed in the future, despite changes happening in both specific package ecosystems and Nixpkgs itself. The Nixpkgs Git history goes back to 2003, offeri… view at source ↗
Figure 2
Figure 2. Overview of the NICE framework for vulnerability reproduction using NixOS. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Workflow for vulnerability reproduction using our NICE framework. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: OpenSSL server VM configuration for Heartbleed [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: NixOS test for Heartbleed vulnerability A key good practice is to keep test parameters coherent: in￾put values used to configure the PoC (e.g., output path and ex￾pected length) 5 are shared with the corresponding assertion block 12 and the exploit invocation 11 . This…
Figure 6
Figure 6. Figure 6: Reproduced timeline of CVEs using NICE framework. The points are scattered randomly on the y-axis for better [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Distribution of CWE-1000 categories across re [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Validation test log of successful Heartbleed reproduction. Part 1: user initialization with [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Validation test log of successful Heartbleed reproduction. Part 2a, showing a [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Validation test log of failed Heartbleed reproduction. Part 2b (alternative to Part 2a in Figure [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 1 canonical work pages

  1. [1]

    https://github.com/minhluannguyen/ nice-archive/tree/preprint-pkg, 2026

    NICE: A Framework for Declarative and Machine- Checkable Vulnerability Reproduction - Replication Package. https://github.com/minhluannguyen/ nice-archive/tree/preprint-pkg, 2026

    2026

  2. [2]

    Reproducible software vulnerability testing with iac

    Kohei Akasaka and Akihito Nakamura. Reproducible software vulnerability testing with iac. In2020 Interna- tional Conference on Computational Science and Com- putational Intelligence (CSCI), pages 36–42, 2020

    2020

  3. [3]

    Unifying package managers, workflow engines, and con- tainers: Computational reproducibility with bionix.Gi- gaScience, 9(11):giaa121, 11 2020

    Justin Bed˝o, Leon Di Stefano, and Anthony T Papenfuss. Unifying package managers, workflow engines, and con- tainers: Computational reproducibility with bionix.Gi- gaScience, 9(11):giaa121, 11 2020

    2020

  4. [4]

    Cybersecurity education and training support system: Cyris.IEICE Transactions on Information and Systems, E101.D(3):740–749, 2018

    Razvan BEURAN, Cuong PHAM, Dat TANG, Ken ichi CHINEN, Yasuo TAN, and Yoichi SHINODA. Cybersecurity education and training support system: Cyris.IEICE Transactions on Information and Systems, E101.D(3):740–749, 2018

    2018

  5. [5]

    Ex- ploitwp2docker: a platform for automating the gener- ation of vulnerable wordpress environments for cyber ranges

    Francesco Caturano, Nicola d’Ambrosio, Gaetano Per- rone, Luigi Previdente, and Simon Pietro Romano. Ex- ploitwp2docker: a platform for automating the gener- ation of vulnerable wordpress environments for cyber ranges. In2022 International Conference on Electrical, Computer and Energy Technologies (ICECET), pages 1–7, 2022

    2022

  6. [6]

    Functional Package Management with Guix

    Ludovic Courtès. Functional package management with guix.CoRR, abs/1305.4584, 2013

    work page Pith review arXiv 2013

  7. [7]

    PhD thesis, Utrecht University, Utrecht, The Netherlands, January 2006

    Eelco Dolstra.The Purely Functional Software Deploy- ment Model. PhD thesis, Utrecht University, Utrecht, The Netherlands, January 2006

    2006

  8. [8]

    Nixos: a purely func- tional linux distribution

    Eelco Dolstra and Andres Löh. Nixos: a purely func- tional linux distribution. InProceedings of the 13th ACM SIGPLAN International Conference on Functional Programming, ICFP ’08, page 367–378, New York, NY , USA, 2008. Association for Computing Machinery

    2008

Show all 32 references
  1. [9]

    Improving reproducibility of scientific software using nix/nixos: A case study on the precice ecosystem.Elec- tronic Communications of the EASST, 83, Feb

    Max Hausch, Simon Hauser, and Benjamin Uekermann. Improving reproducibility of scientific software using nix/nixos: A case study on the precice ecosystem.Elec- tronic Communications of the EASST, 83, Feb. 2025

    2025

  2. [10]

    Intentional repro- duction of vulnerable environments

    Seiya Kamata and Akihito Nakamura. Intentional repro- duction of vulnerable environments. In Hamid R. Arab- nia, Leonidas Deligiannidis, Farzan Shenavarmasouleh, Soheyla Amirian, and Farid Ghareh Mohammadi, edi- tors,Computational Science and Computational Intel- ligence, page...

    2025

  3. [11]

    Sustainable packaging of quantum chemistry software with the nix package manager.International Journal of Quantum Chemistry, 122(9):e26872, 2022

    Markus Kowalewski and Phillip Seeber. Sustainable packaging of quantum chemistry software with the nix package manager.International Journal of Quantum Chemistry, 122(9):e26872, 2022

    2022

  4. [12]

    Le, and David Lo

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D. Le, and David Lo. Refining chatgpt-generated code: Characterizing and mitigating code quality issues.ACM Trans. Softw. Eng. Methodol., 33(5), June 2024

    2024

  5. [13]

    Automated vulnerability validation and verification: A large language model approach, 2025

    Alireza Lotfi, Charalampos Katsis, and Elisa Bertino. Automated vulnerability validation and verification: A large language model approach, 2025

    2025

  6. [14]

    Reproducibility of Build Environments through Space and Time

    Julien Malka, Stefano Zacchiroli, and Théo Zimmer- mann. Reproducibility of Build Environments through Space and Time. InProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineer- ing: New Ideas and Emerging Results, ICSE-NIER’24, pages 97–101, New Yo...

    2024

  7. [15]

    Does Functional Package Management Enable Reproducible Builds at Scale? Yes

    Julien Malka, Stefano Zacchiroli, and Théo Zimmer- mann. Does Functional Package Management Enable Reproducible Builds at Scale? Yes. In22nd Interna- tional Conference on Mining Software Repositories, Ot- tawa, Canada, April 2025

    2025

  8. [16]

    Under- standing the Reproducibility of Crowd-reported Security Vulnerabilities

    Dongliang Mu, Alejandro Cuevas, Limin Yang, Hang Hu, Xinyu Xing, Bing Mao, and Gang Wang. Under- standing the Reproducibility of Crowd-reported Security Vulnerabilities. In27th USENIX Security Symposium (USENIX Security 18), pages 919–936, Baltimore, MD, August 2018. USENIX As...

    2018

  9. [17]

    Cyexec*: Automatic generation of randomized cyber range scenarios

    Ryotaro Nakata and Akira Otsuka. Cyexec*: Automatic generation of randomized cyber range scenarios. In Proceedings of the 7th International Conference on In- formation Systems Security and Privacy - ICISSP, pages 226–236. INSTICC, SciTePress, 2021

    2021

  10. [18]

    Merge the nixos respos- itory into nixpkgs

    NixOS/nixpkgs contributors. Merge the nixos respos- itory into nixpkgs. https://github.com/NixOS/ nixpkgs/commit/2a537fb, 2013. Accessed: 2026-01- 28

    2013

  11. [19]

    DECRET: DEbian CVE Reproducer Tool

    Orange-OpenSource. DECRET: DEbian CVE Reproducer Tool. https://github.com/ Orange-OpenSource/decret, 2023–2026. Ac- cessed: 2026-01-19

    2023

  12. [20]

    Metasploit Framework

    Rapid7. Metasploit Framework. https://github. com/rapid7/metasploit-framework. Accessed: 2026-02-04. 16

    2026

  13. [21]

    Kernjc: Automated vulnerable environment gen- eration for linux kernel vulnerabilities

    Bonan Ruan, Jiahao Liu, Chuqi Zhang, and Zhenkai Liang. Kernjc: Automated vulnerable environment gen- eration for linux kernel vulnerabilities. InProceed- ings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses, RAID ’24, page 384–402, New York...

    2024

  14. [22]

    Automated and reproducible application traces generation for iot applications

    Nina Santi, Rémy Grünblatt, Brandon Foubert, Aroosa Hameed, John Violos, Aris Leivadeas, and Nathalie Mit- ton. Automated and reproducible application traces generation for iot applications. InProceedings of the 17th ACM Symposium on QoS and Security for Wire- less and Mobile ...

    2021

  15. [23]

    Cliffe Schreuders, Thomas Shaw, Mohammad Shan- A-Khuda, Gajendra Ravichandran, Jason Keighley, and Mihai Ordean

    Z. Cliffe Schreuders, Thomas Shaw, Mohammad Shan- A-Khuda, Gajendra Ravichandran, Jason Keighley, and Mihai Ordean. Security scenario generator (SecGen): A framework for generating randomly vulnerable rich- scenario VMs for learning computer security and host- ing CTF events. ...

    2017

  16. [24]

    Pentestagent: Incorporating llm agents to auto- mated penetration testing, 2025

    Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. Pentestagent: Incorporating llm agents to auto- mated penetration testing, 2025

    2025

  17. [25]

    CWE-1000: Research Concepts (4.19.1)

    The MITRE Corporation. CWE-1000: Research Concepts (4.19.1). https://cwe.mitre.org/data/ definitions/1000.html, January 2026. Page last up- dated: January 21, 2026

    2026

  18. [26]

    From cve entries to verifiable exploits: An automated multi-agent framework for reproducing cves, 2025

    Saad Ullah, Praneeth Balasubramanian, Wenbo Guo, Amanda Burnett, Hammond Pearce, Christopher Kruegel, Giovanni Vigna, and Gianluca Stringhini. From cve entries to verifiable exploits: An automated multi-agent framework for reproducing cves, 2025

    2025

  19. [27]

    Vaillancourt, J

    Peter Z. Vaillancourt, J. Eric Coulter, Richard Knepper, and Brandon Barker. Self-scaling clusters and repro- ducible containers to enable scientific computing. In 2020 IEEE High Performance Extreme Computing Con- ference (HPEC), pages 1–8, 2020

    2020

  20. [28]

    Automating system tests using declarative virtual machines

    Sander van der Burg and Eelco Dolstra. Automating system tests using declarative virtual machines. In2010 IEEE 21st International Symposium on Software Relia- bility Engineering, pages 181–190, 2010

    2010

  21. [29]

    Hackers vs

    Daniel V otipka, Rock Stevens, Elissa Redmiles, Jeremy Hu, and Michelle Mazurek. Hackers vs. Testers: A Com- parison of Software Vulnerability Discovery Processes. In2018 IEEE Symposium on Security and Privacy (SP), pages 374–391, 2018

    2018

  22. [30]

    Vulhub: Pre-built vulnerable en- vironments based on docker

    Vulhub contributors. Vulhub: Pre-built vulnerable en- vironments based on docker. GitHub repository, n.d. Accessed 2026-01-20

    2026

  23. [31]

    Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W

    Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Pee- tathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Al...

    2025

  24. [32]

    / nix / store /5 mrdnyhca040405vnbnjsncvqx1 ... sys . exit ( main ()) ^^^^^^ File

    Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. Cve-bench: A benchmark for ai agents’ ability to exploit real-worl...

    2025

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.