Pith. sign in

REVIEW 2 major objections 7 minor 54 references

Frontier LLM agents detect Rust memory-safety bugs at 56–65% but localize the broken lines only around 20% F1, and misleading comments cut that further by about 27%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 14:27 UTC pith:JKI5KMCM

load-bearing objection Solid, release-ready Rust vuln benchmark: multi-level compilable variants plus GT-preserving mutations, with clear agent gaps (detect ~60%, line F1 ~20%, malignant ~27% relative drop). the 2 major comments →

arxiv 2607.04729 v1 pith:JKI5KMCM submitted 2026-07-06 cs.CR cs.AIcs.SE

RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities

Tarek Elsayed , Shiping Yang , Eunsong Koh , Sanika Goyal , Vincent Huang , Paul Ngo , Nathan Young , Mohammad Omidvar Tehrani
show 8 more authors
Alvyn Kang Arnell Kang Zeyu Chen Ang\'elica Moreira Xuan Feng Angel X. Chang Nick Sumner Steven Y. Ko
This is my paper
classification cs.CR cs.AIcs.SE
keywords Rustvulnerability analysisLLM agentscompilable benchmarksdata contaminationline localizationadversarial robustnessCWE classification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing vulnerability benchmarks give models short non-compilable snippets, ask only whether code is vulnerable, and ignore the chance that public datasets have already entered training corpora. RustMizan answers with a Rust-focused framework built on three principles: four analysis tasks (binary detection, CWE class, function location, line location), the same real CVE packaged as three standalone compilable programs at crate, file, and function scope, and an automatic mutation engine that produces semantics-preserving and adversarial variants for contamination and robustness tests. On 42 real memory-safety CVEs turned into 173 compilable variants, four frontier models run as agents with shell access reach only 56–65% binary accuracy while line-level F1 stays near 20%. Narrow function context helps localization; benign mutations leave scores essentially unchanged (no aggregate memorization signal), while adversarial safety cues drop line F1 by roughly 27%. The authors release the dataset, mutation tools, trajectories, and leaderboard so others can regenerate fresh mutants and measure analysis rather than recall.

Core claim

When frontier LLM agents are given interactive shell access to multi-level compilable Rust packages, they can detect memory-safety vulnerabilities at 56–65% accuracy, yet they localize the actual broken lines at only about 20% F1; the same agents are further degraded by roughly 27% relative line-F1 when the code is seeded with misleading safety comments or renames, while semantics-preserving mutations produce no comparable aggregate drop.

What carries the argument

Multi-level compilable variants of the same CVE (crate-, file-, and function-scoped standalone crates) paired with a mutation framework that applies benign, malignant, and Rust-specific transforms while automatically tracking ground-truth function and line labels via markers, content matching, and rename maps.

Load-bearing premise

The ground truth treats every pre-patch version as vulnerable and every post-patch version as clean, assuming each patch fully fixes the disclosed issue and that the reduced variants contain no other bugs.

What would settle it

If models continue to score high line-F1 on freshly mutated variants that preserve semantics but erase surface tokens, or if independent static tools and human auditors disagree with the pre-/post-patch labels on a large fraction of the 173 variants, the claim that current agents cannot localize and that the mutations expose true reasoning failure would not hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. RustMizan is a benchmarking framework for Rust memory-safety vulnerability analysis built from 42 RustSec CVEs packaged as 173 multi-level compilable variants (crate, file, function) with dual-reviewed ground truth for four tasks: binary detection, CWE classification, function localization, and line localization. A paired mutation engine produces benign, malignant, and Rust-specific mutants while preserving ground truth via marker, content, and rename tracking. In an agentic ReAct setup with shell access, four frontier models achieve 56–65% CVC accuracy but only ~17–23% line-localization F1; malignant adversarial cues reduce line F1 by about 27% relative. A controlled SFT contamination study (Appendix G) shows that fine-tuning on vanilla inflates scores that then collapse under benign/malignant mutations, and Kani/RAPx runs (Appendix K) confirm that variants are usable by traditional analyzers.

Significance. If the results hold, the paper supplies a concrete template for vulnerability-analysis benchmarks that other languages can adopt: multi-task evaluation beyond binary detection, multi-granularity compilable context for controlled studies, and built-in contamination/robustness mutations with automated ground-truth maintenance. The combination is well differentiated from snippet-level vulnerability datasets and from agentic coding or repair benchmarks (Table 1, Appendix I). Strengths that should be credited explicitly include: (i) manual dual-review curation rather than automated labeling; (ii) compiler-ecosystem mutations (rustfmt, syn/quote, rust-analyzer) that avoid LLM-in-the-loop mutation; (iii) the controlled SFT contamination experiment validating that the mutations detect memorization when present; (iv) demonstration that the same variants work with Kani and RAPx; and (v) full artifact release (dataset, mutation tool, trajectories, leaderboard). The empirical gap between detection and line localization, and the susceptibility to adversarial cues, are practically useful findings for agent deployment.

major comments (2)
  1. [§3.2–3.5, Tables 3–4] §3.2–3.5, Tables 3–4, Figure 3: Headline claims (56–65% CVC, line F1 near 20%, ~27% relative malignant drop) are reported as point estimates over 173 variants derived from only 42 CVEs. Crate/file/function variants of the same CVE are not independent samples, so treating n=173 as the effective sample size overstates precision. Please report uncertainty (e.g., bootstrap CIs or CVE-level aggregation) and state the unit of analysis clearly so readers can judge how stable the detection–localization gap and the malignant drop are.
  2. [§2.1, §5] §2.1 and §5: Ground truth for all four tasks treats pre-patch code as vulnerable and post-patch as clean. This is standard practice and is flagged in Limitations, but it is load-bearing for every reported metric. The manuscript does not quantify residual risk (incomplete patches, other bugs remaining after reduction, or reachability changes in file/function reductions). A short audit—e.g., spot-check of a subset of patched variants with Kani/RAPx or dual-reviewer residual-bug notes—would substantially strengthen confidence in the absolute scores without changing the framework contribution.
minor comments (7)
  1. [Figure 3] Figure 3 caption and body: the shaded band is described as the gap between average detection and average line F1; state the exact averages used so the figure is self-contained.
  2. [Table 1, Appendix I] Table 1 and Appendix I: the uniqueness claim is carefully hedged (“to the best of our knowledge”), but a one-sentence note on how recently released Rust-focused or agentic security benchmarks were checked would help future readers.
  3. [§3.4] §3.4: Aggregate metrics show no contamination drop, yet trajectory analysis finds Gemini explicitly recalling a CVE. The text already notes CoT faithfulness limits; consider elevating this single case into a short “metric vs. trajectory” caveat in the main RQ3 paragraph so readers do not over-read “no evidence.”
  4. [Appendix D] Appendix D Table 7: several cells are dark gray for invalid JSON; a one-line summary of invalid-JSON rates per model in the main text (or a pointer to Appendix E) would help interpret Success@1 denominators.
  5. [Table 2] §2.2 / Table 2: “No Human in Loop” is accurate for applying mutations, but ground-truth construction is human; a clarifying footnote would avoid misreading the comparison table.
  6. [Abstract] Minor consistency: abstract says “line localization F1 stays near 20%” while Table 3 ranges 17.5–23.2; aligning the abstract wording with the table range would be cleaner.
  7. [§5] §5 mutation-coverage limitation: the per-variant log is mentioned; stating the fraction of variants that received the full benign/malignant suite would make uneven coverage concrete.

Circularity Check

0 steps flagged

No circular derivation: empirical benchmark measurements and externally sourced ground truth, not predictions forced by construction.

full rationale

RustMizan is a dataset/framework paper, not a first-principles derivation. Binary detection (56–65%), line F1 (~20%), and the ~27% relative malignant drop are measured agent outputs scored against dual-reviewed RustSec/CVE annotations, not quantities defined into the metric. Ground truth is external (RustSec advisories, CVE text, commits, dual human consensus), not fitted from the models under test. Mutations are validated by compilation and open-crate test suites with marker/content/rename GT tracking; the controlled SFT contamination study fine-tunes open models on Vanilla and shows drops on mutants—an independent check, not a self-definitional loop. Table 1 uniqueness is a comparative claim about feature coverage, not a uniqueness theorem imported from the authors. Pre-/post-patch labeling (§5) is a load-bearing ground-truth assumption, not circularity. No self-definitional step, fitted-input-as-prediction, load-bearing self-citation chain, or renamed known result appears in the argument.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 2 invented entities

Load-bearing content is mostly domain practice and engineering choices, not free parameters fitted to maximize a physical constant. The central empirical claims rest on human dual-reviewed labels, the pre-/post-patch oracle, semantics-preserving mutations implemented via rustfmt/syn/rust-analyzer, and the ReAct agent protocol. No invented physical entities; the ‘entities’ are the benchmark constructs themselves.

free parameters (2)
  • Mutation operator set and application order (benign / malignant / Rust-specific catalog)
    Which rewrites run and in what sequence is a design choice that defines the contamination and robustness probes; coverage is uneven when constructs (e.g. loops) are absent (§5, Table 5).
  • Agent message/time limits and shared ReAct prompt
    100-message / 3600s limits and a fixed JSON schema (Appendix L.3, F) shape measured performance; not fitted to maximize a claim but still free protocol knobs.
axioms (4)
  • domain assumption Pre-patch variant is vulnerable; post-patch variant is non-vulnerable for binary, CWE, function, and line labels.
    Stated as established practice and listed as a limitation in §5; defines all supervised metrics.
  • domain assumption Compiler-ecosystem mutations (rustfmt, AST rewrites, rust-analyzer renames) preserve program semantics and, with marker/content/rename tracking, preserve ground-truth locations.
    Core of §2.2 and Appendix B; validated via cargo check and external crate test suites, not formal proof of semantic equivalence for every operator.
  • domain assumption RustSec-selected memory-safety CVEs with dual human review yield reliable function- and line-level annotations.
    Curation process in §2.1; quality-over-quantity trade-off acknowledged.
  • ad hoc to paper Set intersection micro-F1 and Success@1 on structured JSON outputs fairly measure agent vulnerability analysis under shell access.
    Metric suite in §3.1 / Appendix C is standard-inspired but specific to this multi-task JSON schema.
invented entities (2)
  • RustMizan multi-level compilable variant family (crate/file/function of the same CVE) independent evidence
    purpose: Enable agentic and traditional-tool evaluation and isolate context granularity effects.
    Constructed artifact of the paper; independent evidence is compilability checks and Kani/RAPx demos, not an external natural phenomenon.
  • Ground-truth-preserving mutation pipeline with marker, content, and rename tracking independent evidence
    purpose: Contamination mitigation and adversarial robustness testing without invalidating line/function labels.
    New engineering stack relative to snippet-only mutation work (Table 2); evidence is compile/test validation and SFT contamination experiment.

pith-pipeline@v1.1.0-grok45 · 34889 in / 3357 out tokens · 31576 ms · 2026-07-11T14:27:34.298295+00:00 · methodology

0 comments
read the original abstract

LLM agents are increasingly applied to vulnerability analysis, but existing benchmarks have not kept pace. They typically rely on small non-compilable snippets, focus on binary classification (vulnerable or not), and do not account for the risk that publicly-released datasets are part of model training corpora. We introduce RustMizan, a benchmarking framework for Rust vulnerability analysis that addresses these gaps. RustMizan contains compilable code variants at the crate, file, and function levels, with annotations for binary vulnerability detection, CWE classification, and function- and line-level localization. A paired mutation framework produces semantics-preserving code mutants for contamination testing and robustness probing. Across four frontier models in an agentic setup with command-line access, binary classification sits in the 56-65% range, but line localization F1 stays near 20%, and adversarial cues drop line F1 by about 27%.

Figures

Figures reproduced from arXiv: 2607.04729 by Alvyn Kang, Ang\'elica Moreira, Angel X. Chang, Arnell Kang, Eunsong Koh, Mohammad Omidvar Tehrani, Nathan Young, Nick Sumner, Paul Ngo, Sanika Goyal, Shiping Yang, Steven Y. Ko, Tarek Elsayed, Vincent Huang, Xuan Feng, Zeyu Chen.

Figure 1
Figure 1. Figure 1: Overview of RUSTMIZAN. (1) a manually curated dataset of 42 RustSec CVEs packaged as 173 multi-level compilable variants with task-specific ground truth, (2) a mutation framework that produces benign, malignant, and Rust-specific code mutants for contamination testing and robustness evaluation, and (3) an LLM-agent evaluation pipeline using a ReAct scaffold with interactive codebase access, scored across f… view at source ↗
Figure 2
Figure 2. Figure 2: Each CVE in RUSTMIZAN is packaged as three standalone compilable crates of decreasing scope: the full crate, a single-file reduction with its compilation dependencies, and a single-function reduction. Red marks the vulnerable file, tracked across all three levels. robustness evaluation. As mentioned earlier, we expect researchers to apply mutations to our vanilla dataset and use the mutated dataset to eval… view at source ↗
Figure 3
Figure 3. Figure 3: Detection and localization scores on RUSTMIZAN (vanilla). All four models detect vulnerabilities at well above chance, but localization scores sit far lower. The shaded band marks the gap between average detection accuracy and average line localization F1 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Line localization F1 across context levels (crate, file, function). Function-level context consistently [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mutation framework pipeline. For each variant, the framework backs up the original code, applies [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Marker tracking through a format-compact mutation. Unique comment markers are inserted before mutation, move with the lines they tag, and are read back to update ground truth before being removed. Content-based tracking (AST mutations). AST-based mutations cannot use marker tracking because the syn and quote crates discard comments during parsing, including the markers. Instead, the framework matches the e… view at source ↗
Figure 7
Figure 7. Figure 7: Example of trajectory-level contamination analysis. Left: An excerpt from Gemini 3.1 Pro’s reasoning [PITH_FULL_IMAGE:figures/full_fig_p032_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 4 canonical work pages

  1. [1]

    ACM Comput

    Sheng, Ze and Chen, Zhicheng and Gu, Shuning and Huang, Heqing and Gu, Guofei and Huang, Jeff , title =. ACM Comput. Surv. , month = nov, articleno =. 2025 , issue_date =. doi:10.1145/3769082 , abstract =

    doi:10.1145/3769082 2025

  2. [2]

    Proceedings of the IEEE/ACM 47th International Conference on Software Engineering , pages =

    Ding, Yangruibo and Fu, Yanjun and Ibrahim, Omniyyah and Sitawarin, Chawin and Chen, Xinyun and Alomair, Basel and Wagner, David and Ray, Baishakhi and Chen, Yizheng , title =. Proceedings of the IEEE/ACM 47th International Conference on Software Engineering , pages =. 2025 , isbn =. doi:10.1109/ICSE55347.2025.00038 , abstract =

    doi:10.1109/icse55347.2025.00038 2025

  3. [3]

    2026 , eprint =

    Rigor, Reliability, and Reproducibility Matter: A Decade-Scale Survey of 572 Code Benchmarks , author =. 2026 , eprint =

    2026

  4. [4]

    2024 , url =

    Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan , booktitle =. 2024 , url =

    2024

  5. [5]

    2024 , eprint =

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author =. 2024 , eprint =

    2024

  6. [6]

    2026 , eprint =

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author =. 2026 , eprint =

    2026

  7. [7]

    Wang, Peiran and Liu, Xiaogeng and Xiao, Chaowei , editor =. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , month = apr, year =. doi:10.18653/v1/2025.naacl-long.212 , pages =

    work page doi:10.18653/v1/2025.naacl-long.212 2025

  8. [8]

    2025 , volume =

    Li, Fengjie and Jiang, Jiajun and Sun, Jiajun and Zhang, Hongyu , booktitle =. 2025 , volume =. doi:10.1109/ICSE-NIER66352.2025.00024 , url =

    doi:10.1109/icse-nier66352.2025.00024 2025

  9. [9]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =

    Wang, Shiqi and Li, Zheng and Qian, Haifeng and Yang, Chenghao and Wang, Zijian and Shang, Mingyue and Kumar, Varun and Tan, Samson and Ray, Baishakhi and Bhatia, Parminder and Nallapati, Ramesh and Ramanathan, Murali Krishna and Roth, Dan and Xiang, Bing , editor =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (...

    doi:10.18653/v1/2023.acl-long.773 2023

  10. [10]

    Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel M. and Wu, Jeffrey and W...

    2020

  11. [11]

    2023 , eprint =

    Detecting Pretraining Data from Large Language Models , author =. 2023 , eprint =

    2023

  12. [12]

    Findings of the Association for Computational Linguistics: ACL 2024 , month = aug, year =

    Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models , author =. Findings of the Association for Computational Linguistics: ACL 2024 , month = aug, year =. doi:10.18653/v1/2024.findings-acl.716 , pages =

    doi:10.18653/v1/2024.findings-acl.716 2024

  13. [13]

    Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , articleno =

    Yang, Zhou and Zhao, Zhipeng and Wang, Chenyu and Shi, Jieke and Kim, Dongsun and Han, Donggyun and Lo, David , title =. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , articleno =. 2024 , isbn =. doi:10.1145/3597503.3639074 , abstract =

    doi:10.1145/3597503.3639074 2024

  14. [14]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , month = dec, year =

    Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , month = dec, year =. doi:10.18653/v1/2023.emnlp-main.308 , pages =

    doi:10.18653/v1/2023.emnlp-main.308 2023

  15. [15]

    arXiv preprint arXiv:2403.07974 , year =

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , author =. arXiv preprint arXiv:2403.07974 , year =

    Pith/arXiv arXiv

  16. [16]

    and Liu, Yong , title =

    Wu, Yonghao and Li, Zheng and Zhang, Jie M. and Liu, Yong , title =. Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering , pages =. 2024 , isbn =. doi:10.1145/3663529.3663815 , abstract =

    doi:10.1145/3663529.3663815 2024

  17. [17]

    Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , pages =

    Wu, Yi and Jiang, Nan and Pham, Hung Viet and Lutellier, Thibaud and Davis, Jordan and Tan, Lin and Babkin, Petr and Shah, Sameena , title =. Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , pages =. 2023 , isbn =. doi:10.1145/3597926.3598135 , abstract =

    doi:10.1145/3597926.3598135 2023

  18. [18]

    and Yanjie, Zhao

    Hongzhou, Rao. and Yanjie, Zhao. and Wenjie, Zhu. and Ling, Xiao. and Meizhen, Wang. and Haoyu, Wang. , title =. Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings , status =

    2025

  19. [19]

    2018 , volume =

    Yeom, Samuel and Giacomelli, Irene and Fredrikson, Matt and Jha, Somesh , booktitle =. 2018 , volume =. doi:10.1109/CSF.2018.00027 , url =

    doi:10.1109/csf.2018.00027 2018

  20. [20]

    2024 , volume =

    Ullah, Saad and Han, Mingji and Pujar, Saurabh and Pearce, Hammond and Coskun, Ayse and Stringhini, Gianluca , booktitle =. 2024 , volume =. doi:10.1109/SP54263.2024.00210 , url =

    doi:10.1109/sp54263.2024.00210 2024

  21. [21]

    ACM Trans

    Zheng, Xiaoye and Wan, Zhiyuan and Zhang, Yun and Chang, Rui and Lo, David , title =. ACM Trans. Softw. Eng. Methodol. , month = dec, articleno =. 2023 , issue_date =. doi:10.1145/3624738 , abstract =

    doi:10.1145/3624738 2023

  22. [22]

    , title =

    Xu, Hui and Chen, Zhuangbin and Sun, Mingshen and Zhou, Yangfan and Lyu, Michael R. , title =. ACM Trans. Softw. Eng. Methodol. , month = sep, articleno =. 2021 , issue_date =. doi:10.1145/3466642 , abstract =

    doi:10.1145/3466642 2021

  23. [23]

    Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation , pages =

    Qin, Boqin and Chen, Yilun and Yu, Zeming and Song, Linhai and Zhang, Yiying , title =. Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation , pages =. 2020 , isbn =. doi:10.1145/3385412.3386036 , abstract =

    doi:10.1145/3385412.3386036 2020

  24. [24]

    2022 , isbn =

    VanHattum, Alexa and Schwartz-Narbonne, Daniel and Chong, Nathan and Sampson, Adrian , title =. 2022 , isbn =. doi:10.1145/3510457.3513031 , booktitle =

    doi:10.1145/3510457.3513031 2022

  25. [25]

    2025 , url =

    Artisan-Lab , title =. 2025 , url =

    2025

  26. [26]

    Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , pages =

    Evans, Ana Nora and Campbell, Bradford and Soffa, Mary Lou , title =. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , pages =. 2020 , isbn =. doi:10.1145/3377811.3380413 , abstract =

    doi:10.1145/3377811.3380413 2020

  27. [27]

    Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , pages =

    Liu, Peiming and Zhao, Gang and Huang, Jeff , title =. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , pages =. 2020 , isbn =. doi:10.1145/3377811.3380325 , abstract =

    doi:10.1145/3377811.3380325 2020

  28. [28]

    AI Security Institute, UK , title =

  29. [29]

    2023 , eprint =

    ReAct: Synergizing Reasoning and Acting in Language Models , author =. 2023 , eprint =

    2023

  30. [30]

    2025 , month =

    Meng, Kevin and Huang, Vincent and Steinhardt, Jacob and Schwettmann, Sarah , title =. 2025 , month =

    2025

  31. [31]

    2025 , eprint =

    Reasoning Models Don't Always Say What They Think , author =. 2025 , eprint =

    2025

  32. [32]

    2025 , howpublished =

    Rust-Lang , title =. 2025 , howpublished =

    2025

  33. [33]

    2021 , howpublished =

    MIRAI Contributors , title =. 2021 , howpublished =

    2021

  34. [34]

    ACM Trans

    Cui, Mohan and Chen, Chengjun and Xu, Hui and Zhou, Yangfan , title =. ACM Trans. Softw. Eng. Methodol. , month = may, articleno =. 2023 , issue_date =. doi:10.1145/3542948 , abstract =

    doi:10.1145/3542948 2023

  35. [35]

    Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles , pages =

    Bae, Yechan and Kim, Youngsuk and Askar, Ammar and Lim, Jungwon and Kim, Taesoo , title =. Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles , pages =. 2021 , isbn =. doi:10.1145/3477132.3483570 , abstract =

    doi:10.1145/3477132.3483570 2021

  36. [36]

    , title =

    Li, Zhuohua and Wang, Jincheng and Sun, Mingshen and Lui, John C.S. , title =. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security , pages =. 2021 , isbn =. doi:10.1145/3460120.3484541 , abstract =

    doi:10.1145/3460120.3484541 2021

  37. [37]

    Proceedings of the 17th European Workshop on Systems Security , pages =

    Louka, Antonis and Dionysiou, Antreas and Athanasopoulos, Elias , title =. Proceedings of the 17th European Workshop on Systems Security , pages =. 2024 , isbn =. doi:10.1145/3642974.3652281 , abstract =

    work page doi:10.1145/3642974.3652281 2024

  38. [38]

    2025 , eprint =

    deepSURF: Detecting Memory Safety Vulnerabilities in Rust Through Fuzzing LLM-Augmented Harnesses , author =. 2025 , eprint =

    2025

  39. [39]

    Li, Huan and Wang, Bei and Hu, Xing and Xia, Xin , title =. Proc. ACM Softw. Eng. , month = jun, articleno =. 2025 , issue_date =. doi:10.1145/3728890 , abstract =

    work page doi:10.1145/3728890 2025

  40. [40]

    2025 , url =

    Anirudh Khatry and Robert Zhang and Jia Pan and Ziteng Wang and Qiaochu Chen and Greg Durrett and Isil Dillig , booktitle =. 2025 , url =

    2025

  41. [41]

    2025 , url =

    Yuxuan Zhu and Antony Kellermann and Dylan Bowman and Philip Li and Akul Gupta and Adarsh Danda and Richard Fang and Conner Jensen and Eric Ihli and Jason Benn and Jet Geronimo and Avi Dhir and Sudhit Rao and Kaicheng Yu and Twm Stone and Daniel Kang , booktitle =. 2025 , url =

    2025

  42. [42]

    CyberGym: Evaluating

    Zhun Wang and Tianneng Shi and Jingxuan He and Matthew Cai and Jialin Zhang and Dawn Song , booktitle =. CyberGym: Evaluating. 2026 , url =

    2026

  43. [43]

    Hwiwon Lee and Ziqi Zhang and Hanxiao Lu and Lingming Zhang , booktitle =

  44. [44]

    Benchmarking

    Yildiz, Alperen and Teo, Sin G and Lou, Yiling and Feng, Yebo and Wang, Chong and Divakaran, Dinil Mon , editor =. Benchmarking. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =. doi:10.18653/v1/2025.acl-long.1490 , pages =

    doi:10.18653/v1/2025.acl-long.1490 2025

  45. [45]

    Daoguang Zan and Zhirong Huang and Wei Liu and Hanwu Chen and Shulin Xin and Linhao Zhang and Qi Liu and Aoyan Li and Lu Chen and Xiaojian Zhong and Siyao Liu and Yongsheng Xiao and Liangqiang Chen and Yuyu Zhang and Jing Su and Tianyu Liu and RUI LONG and Ming Ding and liang xiang , booktitle =. Multi-. 2026 , url =

    2026

  46. [46]

    2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE '26) , year =

    Xiang, Jiahong and He, Wenxiao and Wang, Xihua and Tian, Hongliang and Zhang, Yuqun , title =. 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE '26) , year =. doi:10.1145/3744916.3773108 , isbn =

    doi:10.1145/3744916.3773108 2026

  47. [47]

    Forty-second International Conference on Machine Learning , year =

    DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination , author =. Forty-second International Conference on Machine Learning , year =

  48. [48]

    , title =

    Fan, Jiahao and Li, Yi and Wang, Shaohua and Nguyen, Tien N. , title =. 2020 , isbn =. doi:10.1145/3379597.3387501 , booktitle =

    doi:10.1145/3379597.3387501 2020

  49. [49]

    2021 , isbn =

    Bhandari, Guru and Naseer, Amara and Moonen, Leon , title =. 2021 , isbn =. doi:10.1145/3475960.3475985 , booktitle =

    doi:10.1145/3475960.3475985 2021

  50. [50]

    2021 , isbn =

    Nikitopoulos, Georgios and Dritsa, Konstantina and Louridas, Panos and Mitropoulos, Dimitris , title =. 2021 , isbn =. doi:10.1145/3468264.3473122 , booktitle =

    doi:10.1145/3468264.3473122 2021

  51. [51]

    Proceedings of the 33rd International Conference on Neural Information Processing Systems , articleno =

    Zhou, Yaqin and Liu, Shangqing and Siow, Jingkai and Du, Xiaoning and Liu, Yang , title =. Proceedings of the 33rd International Conference on Neural Information Processing Systems , articleno =. 2019 , publisher =

    2019

  52. [52]

    2023 , isbn =

    Chen, Yizheng and Ding, Zhoujie and Alowain, Lamya and Chen, Xinyun and Wagner, David , title =. 2023 , isbn =. doi:10.1145/3607199.3607242 , booktitle =

    doi:10.1145/3607199.3607242 2023

  53. [53]

    Quantifying Memorization Advantage in Code LLMs , booktitle =

    Euraste, Djiré and Kader, Kaboré and Samhi, Jordan and Barr, Earl and Klein, Jacques and Bissyandé, Tegawendé , year =. Quantifying Memorization Advantage in Code LLMs , booktitle =

  54. [54]

    doi:10.5281/zenodo.19488581 , url =

    Ji, Zhengjie and Wang, Xin and Wang, Lingxiang and Li, Geng and Yang, Fan and Zhang, Ying , title =. doi:10.5281/zenodo.19488581 , url =

    work page doi:10.5281/zenodo.19488581