Pith. sign in

REVIEW 6 minor 39 references

EduPluginBench: Executable Assurance for AI-Generated Educational Plugins

T0 review · 0 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Executable contracts lift plugin-defect recall from 25% to 100%.

desk verdict A carefully constructed benchmark whose headline recall is a contract-consistency result—valuable as a reproducibility model and for its honest negative-transfer evidence, but not yet a demonstration of real-defect detection. read the letter →

arxiv 2608.00739 v1 pith:Z7437WIQ submitted 2026-08-01 cs.SE cs.AI

Nizam Kadir This is my paper · ORCID
classification cs.SEcs.AI
keywords AI-generatedcodesoftwareassurancemutationtestingpolicyassupplychaineducationalpluginsexecutablecontracts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EduPluginBench asks whether an AI-generated plugin should be admitted to a governed ecosystem, not just whether it runs. The paper builds a staged P0-P4 assurance pipeline that layers static security analysis, supply-chain policy, executable domain contracts, and sandboxed behavioral checks over ordinary build and functional tests, then measures it on 1,440 single-defect mutant packages, each constructed and activation-checked from one of 30 educational-plugin specifications. On that controlled benchmark, the full pipeline raises release-blocking-defect recall from 25.3% to 100% (specification-clustered 95% CI 73.4-75.8 percentage points) while rejecting none of 120 clean references (95% Wilson upper bound 3.1%). The paper is careful to claim only consistency with the encoded contract, not independent real-defect discovery: a separately frozen transfer study of 600 unmodified model-generated packages found none that passed even the first stage, and a generic PHP ruleset detected none of 16 independently labelled vulnerable/fixed pairs from a public learning-platform security-advisory corpus. A sympathetic reader would care because plugin platforms need release decisions that span code, dependencies, runtime behavior, and domain policy, and this paper supplies a concrete executable method with its limits stated.

What carries the argument

The carrying mechanism is a staged fail-closed pipeline: P0 (package schema, build, visible functional tests, deterministic replay), P1 (pinned Bandit and Semgrep static rules), P2 (lockfile, CycloneDX SBOM, and a frozen advisory snapshot), P3 (executable domain contracts over capability, telemetry, provenance, lifecycle, resource, and accessibility records), and P4 (network-isolated execution with metamorphic replay and behavioral-record checks). Its validity instrument is the mutation registry: 40 first-order operators, five per family, each applied only after a label-separated construction oracle confirms activation and isolation, giving 1,440 valid mutants whose decisions are the primary recall denominator. The two rejection-bearing P4 relations are that an irrelevant event field must not change the result and that the returned evidence version and retrieval time must track the stale-evidence input; remaining probes stay diagnostic.

What would settle it

Run the full P0-P4 pipeline on an independently authored corpus of real educational-plugin vulnerabilities with black-box labels and a fresh specification schema; if recall is materially below 100% or clean-package rejections exceed the 3.1% Wilson bound, the contract-consistency claim is falsified. A cheaper check: strip mutation identifiers and lineage labels from package objects and verify that every P3/P4 decision is unchanged; any decision flip would break the label-separation claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that release-blocking defects in plugin packages split into eight families—security, dependency, privacy, authority, provenance, lifecycle, reliability, and accessibility—and that after build, static-security, and dependency stages catch the first two, executable domain contracts (P3) and behavioral-record and metamorphic runtime checks (P4) catch the remaining six. On the constructed benchmark, P0-P2 alone recalled 25.3% of 1,440 activation-validated first-order mutants, while P0-P4 recalled 100%, an absolute 74.7-percentage-point difference; P3 uniquely detected 680 mutants and P4 uniquely detected 395. No clean reference was rejected. The paper states that this establishes consistency with the encoded contract, not independent production-defect effectiveness, and the transfer strata confirm that boundary: 300 of 600 unmodified generations parsed but 0 of 600 passed P0, and the frozen generic PHP detector found 0 of 16 vulnerable revisions in the external security pairs.

Load-bearing premise

The load-bearing premise is that the 1,440 activation-checked first-order mutants are valid release-blocking defects, where activation is decided by author-written construction oracles that share the specification vocabulary and public manifest or behavioral fields with the P3/P4 detectors; if those mutants are not representative of real defects, the 100% recall measures consistency with the benchmark's own encoding rather than real-world detection.

Editorial extensions

If this is right

  • Plugin platforms can express admission obligations as versioned executable contracts instead of prose review criteria, covering least privilege, telemetry consent, provenance, lifecycle, and bounded failure.
  • Conventional build, static-security, and dependency checks alone leave six of eight defect families undetected in the controlled stratum; the contract and behavioral stages are what recover them.
  • If the benchmark is right, current model-generated plugins fail at interface and visible-functionality gates (0 of 600 unmodified generations passed P0), so downstream assurance estimates remain undefined without repair or better generation.
  • Generic security rulesets do not transfer to application-specific vulnerabilities: the frozen PHP ruleset found none of 16 independently labelled vulnerable/fixed pairs.
  • Clean-package harm was not observed in the controlled stratum, but the 95% Wilson upper bound of 3.1% means a nonzero rejection rate is not excluded.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the zero-P0 transfer results generalize, the practical bottleneck for admitting generated plugins is interface and functional compliance rather than detector sensitivity; even perfect P3-P4 detectors cannot be exercised on packages that never form valid releases.
  • The shared vocabulary between construction oracles and detectors means the 100% recall is a consistency result within the benchmark's own encoding; an independently authored corpus with black-box labels is the natural next experiment and would likely show lower recall.
  • The porting recipe generalizes beyond education to other governed plugin ecosystems, such as enterprise workflow or scientific analysis, but the specific effect sizes should not be carried over without rebuilding the specification schema, clean corpus, and domain oracles.
  • The external zero-detection result suggests that governance needs application-aware contracts rather than generic language rulesets; training and evaluating a domain-specific detector on the same public vulnerable/fixed pairs would be a testable follow-up.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 6 minor

Summary. The paper introduces EduPluginBench, an executable benchmark and a staged P0–P4 admission pipeline for AI-generated educational plugins. The benchmark defines eight defect families, 30 specifications, 120 clean reference packages, and 1,440 activation-checked first-order mutants. The controlled evaluation reports that P0–P4 increases mutant recall from 25.3% under P0–P2 to 100%, a 74.7 percentage-point difference, with 0/120 observed clean-package rejections. The paper also reports three negative transfer results: 0/600 unmodified model-generated candidates pass P0; a frozen generic PHP detector finds 0/16 Moodle vulnerable revisions; and an earlier bounded-repair diagnostic is presented as a separate, clearly post hoc stratum. The authors explicitly frame the controlled result as consistency with the encoded contract, not independent real-defect effectiveness, and they provide a checksummed artifact with row-level data, frozen protocols, and reproduction instructions.

Significance. The paper's main strengths are its unusually explicit evidence hierarchy, protocol locking, specification-clustered inference, label-separation checks, and the clear separation of confirmatory, post hoc, and scenario analyses. The negative transfer studies are a real contribution: they sharply delimit what the benchmark can and cannot claim. The central caveat, stated in Section 9.1 and in the conclusion, is that the construction oracles and the P3/P4 detectors operationalize the same specification vocabulary, so the 74.7-point recall gain is a contract-consistency result rather than evidence of independent real-defect discovery. Because the paper is transparent about this and scopes its claims accordingly, the benchmark remains useful as a controlled, reproducible instrument for measuring contract consistency and clean-package harm, even though it does not by itself establish real-defect detection effectiveness.

minor comments (6)
  1. [§7.5] The sentence "Under the frozen parser, 50.0% (300/600; 95% CI 46.01–53.99%)" is missing the noun after the percentage; it should say "50.0% parsed" or "yielded a parseable package."
  2. [§9.1] Because the paper itself states that oracles and detectors are not construct-independent, the abstract should consider renaming "release-blocking-defect recall" to "controlled contract-consistency recall" at first mention, or adding a one-sentence qualifier immediately after the 74.7 percentage-point number, so that a casual reader does not take the figure as real-defect recall.
  3. [§7.1] Given that all 4,800 operator–parent pairs were applicable, activated, and isolated, the paper should state explicitly whether the activation oracle was designed to be total, so that the 100% activation rate is not read as empirical evidence of mutant representativeness.
  4. [§5.6] The claim that P6 satisfies the preregistered 10% clean-package rejection bound on held-out specifications rests on a 0/20 point estimate whose Wilson upper bound is 16.1%; the paper should phrase this as observed satisfaction and avoid implying the bound is statistically confirmed.
  5. [§7.5] The model-specific S6 result would be clearer if the abstract or results section explicitly stated that all 300 parseable outputs came from a single model and all 300 interface failures came from the other model, since this split is central to interpreting the 50.0% parse yield.
  6. [§8.4] The sentence "The extent to which each recommendation is supported is summarized by the controlled and bounded-repair evidence supports executable contracts..." is grammatically tangled and should be rewritten.

Circularity Check

3 steps flagged · score 6.0 of 10

The headline recall gain and clean-reference rejection are by-construction contract-consistency results: mutant validity, clean selection, and P3/P4 detectors all operationalize the same family oracles, and the paper explicitly says so.

  1. self definitional [Section 3.3 'Contract families' and Eq. (1) in Section 4.3 'Mutation registry']
    "For a parent p and operator μ, a mutant p′=μ(p) is valid only when: H_f(p,s)=0, H_f(p′,s)=1 … The oracles and detectors are not construct-independent because both operationalize the same specification and can inspect the same public manifest or behavioral field."

    The primary controlled-stratum denominator is restricted to mutants whose construction oracle H_f flips. P3/P4 detectors are executable implementations of the same eight family definitions (Table 3 and Section 5.4), so any correct detector must reject all 1,440 valid mutants. The 74.7-percentage-point recall gain is therefore entailed by the mutant-inclusion rule plus detector correctness; it is a consistency property of the encoding, not an empirical discovery. The label-separation tests in Section 9.1 only show detectors do not read identifiers or labels; they do not break the shared-contract identity. The paper's own wording, 'consistency with the encoded contract, not independent production-defect effectiveness', concedes this reduction.

  2. self definitional [Section 4.2 'Clean reference packages']
    "A package is accepted as clean only if it passes visible functional tests, the hidden conformance suite, all family oracles, deterministic replay, dependency policy, and a duplicate-digest check."

    Clean references enter the false-positive denominator only after passing all family oracles, and P3/P4 evaluate the same contract families. Zero rejections among 120 clean references is therefore a selection consequence rather than an independent measurement of false-positive behavior. The paper acknowledges this by describing the clean packages as instantiations of four author-designed source families and by reporting only a Wilson upper bound, but the 0/120 result is nevertheless forced by the clean-acceptance definition.

1 more flagged steps
  1. self definitional [Section 4.4 'Model-generated candidates']
    "Their family labels derive from reference-oracle functions that are label-separated from the detector stages but not construct-independent. For properties represented in manifest or behavioral records, the reference oracle and detector necessarily inspect some of the same public fields."

    The bounded-repair generated stratum reports 112/112 P0-passing candidates as nonconforming and then finds P0-P4 recall of 100% on the same stratum. Since the nonconformance labels come from the same reference-oracle family that P3/P4 implement, the perfect recall is again by construction. The paper explicitly calls this 'corroboration, not as construct-independent ground truth', so it cannot serve as external validation of the controlled-mutant result; it is another instance of the same contract-consistency identity.

full rationale

The paper is unusually transparent about the boundary between constructed consistency and independent effectiveness: it repeatedly states that perfect detection of activation-validated mutants establishes consistency with the encoded contract, and its two transfer studies (S6 and S7) are genuinely independent and negative. The primary recall estimate is nonetheless structurally circular in the sense that matters for this review: a mutant is valid only when the family oracle flips, and P3/P4 detectors are executable versions of the same family definitions, so 100% recall is implied by the inclusion criterion plus detector correctness. Similarly, clean references are selected by passing all family oracles, so 0/120 clean rejections is by construction. The bounded-repair generated stratum repeats the same pattern with oracle-derived labels. These are not hidden circularities; the paper names them as contract-consistency checks and declines to generalize them to real-defect effectiveness. No load-bearing self-citation was found: the LearnAdapt citation is used only to motivate the domain, and the paper states no data or code from that work is reused. The negative transfer results (0/600 unmodified P0 passes in S6; 0/16 Moodle vulnerabilities found in S7) are independent evidence and correctly prevent the controlled result from being read as real-world effectiveness. Because the central quantitative claims reduce by construction, a moderate circularity score is appropriate, but the paper's own honesty and external negative results keep it below the highest levels.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central effectiveness claim rests on author-defined mutation operators and construction oracles that share their vocabulary with the detectors; no external real-defect corpus is used to validate the controlled-mutant stratum. The statistical methods are standard; the domain assumptions are declared.

free parameters (3)
  • P6 stage weights = 12, 5, 4, 3, 2 for P0-P4
    Chosen to encode the release scenario in Section 5.6, not derived; threshold calibration depends on them.
  • Defect-family severity weights = 5 for security/privacy/authority; 4 for dependency/provenance/lifecycle; 3 for reliability/accessibility
    Ordinal weights fixed in the protocol and used in P6 calibration and queue metrics (Section 5.6).
  • P6 admission thresholds = quarantine at score 1, reject at score 2
    Selected by enumeration over construction/calibration data subject to the harm bounds; not retuned on held-out rows (Section 7.8).
assumptions (5)
  • domain assumption The eight defect families operationalize release-blocking defects for governed plugin ecosystems.
    Declared in Sections 3.2 and 3.3; the paper notes this is not exhaustive.
  • domain assumption Activation-checked first-order mutants represent release-blocking defects in generated code, i.e., the mutation coupling effect holds.
    Section 2.3 cites mutation testing literature but explicitly says validity and representativeness require evidence; the paper does not validate against real faults.
  • domain assumption The construction oracle H_f correctly identifies the intended defect family and isolation.
    Validity equations (1) and (2) in Section 4.3; oracle correctness is assumed, and Section 9.1 notes oracle/detector construct-sharing.
  • domain assumption Specifications are representative of educational plugin tasks.
    30 specifications across 5 archetypes; Section 9.4 limits external validity.
  • standard math Cluster bootstrap and Wilson intervals give valid uncertainty for the designed corpus.
    Statistical methods described in Section 6.6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EduPluginBench: Executable Assurance for AI-Generated Educational Plugins." pith.science (2026). https://pith.science/paper/Z7437WIQ

@misc{pith2026260800739,
  author       = {Pith},
  title        = {Pith review of: EduPluginBench: Executable Assurance for AI-Generated Educational Plugins},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7437WIQ}},
  note         = {Machine review of arXiv:2608.00739}
}
read the original abstract

Code-generation models can produce executable components, but compilation and functional tests do not establish compliance with least privilege, telemetry consent, provenance, privileged-write authority, lifecycle constraints, or bounded failure. We introduce EduPluginBench, an executable benchmark and staged admission method for generated plugins in governed software ecosystems. Across 1,440 activation-checked first-order mutants from 30 specifications, P0-P4 increased release-blocking-defect recall by 74.7 percentage points (specification-clustered 95% CI 73.4-75.8) over P0-P2, with no observed rejection among 120 clean references (95% Wilson upper bound 3.1%). A frozen transfer study of 600 unmodified generations from two current coding models found that 300/600 parsed, but none passed P0 or achieved P0-P4 conformance (95% upper bound 0.64%); downstream assurance estimands were undefined. An independently labelled Moodle study retained 16 vulnerable/fixed pairs; the frozen generic PHP detector found no vulnerable revisions. These negative transfer results prevent controlled contract consistency from being read as independent real-defect effectiveness. An earlier 540-generation diagnostic found that post-hoc bounded repair yielded 112 P0 passes, all nonconforming, with recall increasing from 13.4% to 100%. The artifact retains protocols, public-source provenance, raw generations, row-level decisions, audits, analysis code, and reproduction instructions.

Figures

Figures reproduced from arXiv: 2608.00739 by the authors.

Figure 1
Figure 1. Stage-specific rejection rates on controlled mutants. Cells show the proportion rejected at that stage, [PITH_FULL_IMAGE:figures/full_fig_p017_1.png] view at source ↗
Figure 2
Figure 2. Primary and post-hoc generation yields. The strict bare-JSON result is the primary endpoint. Hatched [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figure 3
Figure 3. Paired-trace queue trade-off. Each marker reports mean clean-package and defective-package waiting [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 19 canonical work pages

  1. [1]

    1EdTech Consortium. 2019. Learning Tools Interoperability Core Specification 1.3. Accessed 2026-07-31. https: //www.imsglobal.org/spec/lti/v1p3

    work page 2019

  2. [2]

    1EdTech Consortium. 2021. Security Framework 1.1. Accessed 2026-07-31. https://www.1edtech.org/standards/ security-framework

    work page 2021

  3. [3]

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, et al

    work page

  4. [4]

    David Basin, Jürgen Doser, and Torsten Lodderstedt. 2006. Model Driven Security: From UML Models to Access Control Infrastructures.ACM Transactions on Software Engineering and Methodology15, 1 (2006), 39–91. doi:10.1145/ 1125808.1125810

    work page arXiv 2006

  5. [5]

    Andreas Bauer, Martin Leucker, and Christian Schallhart. 2011. Runtime Verification for LTL and TLTL.ACM Transactions on Software Engineering and Methodology20, 4, Article 14 (2011), 64 pages. doi:10.1145/2000799.2000800

    arXiv 2011

  6. [6]

    Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, et al. 2023. Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models. arXiv:2312.04724 [cs.CR] doi:10.48550/arXiv.2312.04724

    arXiv doi:10.48550/arxiv.2312.04724 2023

  7. [7]

    Tingting Bi, Xin Xia, David Lo, John Grundy, Thomas Zimmermann, and Denae Ford. 2022. Accessibility in Software Practice: A Practitioner’s Perspective.ACM Transactions on Software Engineering and Methodology31, 4, Article 66 (2022), 26 pages. doi:10.1145/3503508

    doi:10.1145/3503508 2022

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, et al . 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] doi:10.48550/arXiv.2107.03374

    arXiv doi:10.48550/arxiv.2107.03374 2021

Show all 39 references
  1. [9]

    Qirui Chen, Jingxian Shuai, Shuangwu Chen, Shenghao Ye, Zijian Wen, Xufei Su, Jie Jin, Jiangming Li, Jun Chen, Xiaobin Tan, and Jian Yang. 2026. HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation. arXiv:2601.13864 [cs.CR] doi:10.48550/arXiv....

    2026 doi arXiv

  2. [10]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-Collaboration Code Generation via ChatGPT.ACM Transactions on Software Engineering and Methodology33, 7, Article 189 (2024), 38 pages. doi:10.1145/3672459

    2024 doi

  3. [11]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming—The Rise of Code Intelligence. arXiv:2401.14196 [cs.SE] doi:10.48550/ arXiv.2401.14196

    2024 doi arXiv

  4. [12]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, et al. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186 [cs.CL] doi:10.48550/arXiv.2409.12186

    2024 doi arXiv

  5. [13]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InInternational Conference on Learning Representations. OpenReview.net, Vienna, Austria, 51 pages....

    2024 doi arXiv

  6. [14]

    René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. InProceedings of the 2014 International Symposium on Software Testing and Analysis. Association for Computing Machinery, New...

    2014

  7. [15]

    Nizam Kadir. 2026. From Tools to Teacher-Built Teammates: No-Code Pedagogical Plugin Authoring with LearnAdapt Agentic Studio and PedOS 1.1 Lumina. arXiv:2607.09674 [cs.HC] doi:10.48550/arXiv.2607.09674

    2026 doi arXiv

  8. [16]

    Thomas Laurent, Mike Papadakis, Marinos Kintis, Christopher Henard, Yves Le Traon, and Anthony Ventresque. 2017. Assessing and Improving the Mutation Testing Practice of PIT. In2017 IEEE International Conference on Software Testing, Verification and Validation. IEEE, Piscatawa...

    2017 doi

  9. [17]

    Le, and David Lo

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D. Le, and David Lo

  10. [18]

    Mistral AI. 2025. Devstral-Small-2507 Model Card. Hugging Face model repository. Revision bd165ab26cebbcc2eea2c4ecbfc07f3ac42b3c39; accessed 2026-08-01. https://huggingface.co/mistralai/Devstral-Small- 2507/tree/bd165ab26cebbcc2eea2c4ecbfc07f3ac42b3c39

    2025

  11. [19]

    Moodle. 2026. Security Announcements. Moodle.org. MSA-26-0012 through MSA-26-0029 and linked public source histories; accessed 2026-08-01. https://moodle.org/security/advisory

    2026

  12. [20]

    Jefferson Offutt

    A. Jefferson Offutt. 1992. Investigations of the Software Testing Coupling Effect.ACM Transactions on Software Engineering and Methodology1, 1 (1992), 5–20. doi:10.1145/125489.125473

    1992

  13. [21]

    Jefferson Offutt, Ammei Lee, Gregg Rothermel, Roland H

    A. Jefferson Offutt, Ammei Lee, Gregg Rothermel, Roland H. Untch, and Christian Zapf. 1996. An Experimental Determination of Sufficient Mutant Operators.ACM Transactions on Software Engineering and Methodology5, 2 (1996), 99–118. doi:10.1145/227607.227610

    1996

  14. [22]

    Open Source Security Foundation. 2026. Supply-chain Levels for Software Artifacts Specification. Accessed 2026-07-31. https://slsa.dev/spec/

    2026

  15. [23]

    Zhang, Mark Harman, and Meng Wang

    Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. 2025. An Empirical Study of the Non-Determinism of ChatGPT in Code Generation.ACM Transactions on Software Engineering and Methodology34, 2, Article 42 (2025), 28 pages. doi:10.1145/3697010

    2025 doi

  16. [24]

    OWASP Foundation. 2024. CycloneDX Bill of Materials Standard, Version 1.6. Accessed 2026-07-31. https://cyclonedx. org/specification/overview/

    2024

  17. [25]

    Ruwei Pan, Yakun Zhang, Qingyuan Liang, Yueheng Zhu, Chao Liu, Lu Zhang, and Hongyu Zhang. 2026. Toward Functional and Non-Functional Evaluation of Application-Level Code Generation. arXiv:2602.03462 [cs.SE] doi:10. 48550/arXiv.2602.03462

    2026 doi

  18. [26]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In2022 IEEE Symposium on Security and Privacy. IEEE, Los Alamitos, CA, USA, 754–768. doi:10.1109/SP...

    2022

  19. [27]

    Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. CWEval: Outcome-Driven Evaluation on Functionality and Security of LLM Code Generation. arXiv:2501.08200 [cs.SE] doi:10.48550/arXiv.2501.08200

    2025 doi arXiv

  20. [28]

    Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023. Do Users Write More Insecure Code with AI Assistants?. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. Association for Computing Machinery, New York, NY, USA, 2785–2799. d...

    2023

  21. [29]

    Qwen Team. 2025. Qwen3-Coder-30B-A3B-Instruct Model Card. Hugging Face model repository. Revision b2cff646eb4bb1d68355c01b18ae02e7cf42d120; accessed 2026-08-01. https://huggingface.co/Qwen/Qwen3-Coder- 30B-A3B-Instruct/tree/b2cff646eb4bb1d68355c01b18ae02e7cf42d120

    2025

  22. [30]

    Semgrep. 2026. Semgrep Community PHP Ruleset and Command-Line Scanner. Semgrep Registry and documentation. Ruleset archived before execution with Semgrep 1.172.0; accessed 2026-08-01. https://semgrep.dev/c/p/php

    2026

  23. [31]

    Chihao Shen, Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2025. SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories. arXiv:2504.21205 [cs.SE] doi:10.48550/arXiv.2504.21205

    2025 doi

  24. [32]

    2022.Secure Software Development Framework (SSDF) Version 1.1

    Murugiah Souppaya, Karen Scarfone, and Donna Dodson. 2022.Secure Software Development Framework (SSDF) Version 1.1. Technical Report NIST SP 800-218. National Institute of Standards and Technology. doi:10.6028/NIST.SP.800-218

    2022 doi

  25. [33]

    Peiding Wang, Li Zhang, Fang Liu, Lin Shi, Minxiao Li, Bo Shen, and An Fu. 2025. CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation. arXiv:2503.22688 [cs.SE] doi:10.48550/arXiv.2503.22688

    2025 doi

  26. [34]

    Yanlin Wang, Ziyao Zhang, Chong Wang, Xinyi Xu, Mingwei Liu, Yong Wang, Jiachi Chen, and Zibin Zheng

  27. [35]

    World Wide Web Consortium. 2023. Web Content Accessibility Guidelines (WCAG) 2.2. W3C Recommendation, accessed 2026-07-31. https://www.w3.org/TR/WCAG22/

    2023

  28. [36]

    Xu, Bogdan Vasilescu, and Graham Neubig

    Frank F. Xu, Bogdan Vasilescu, and Graham Neubig. 2022. In-IDE Code Generation from Natural Language: Promise and Challenges.ACM Transactions on Software Engineering and Methodology31, 2, Article 29 (2022), 47 pages. doi:10.1145/3487569

    2022 doi

  29. [2024]

    doi:10.1145/3643674 EduPluginBench: Executable Assurance for AI-Generated Educational Plugins 27

    Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues.ACM Transactions on Software Engineering and Methodology33, 5, Article 116 (2024), 26 pages. doi:10.1145/3643674 EduPluginBench: Executable Assurance for AI-Generated Educational Plugins 27

    2024 doi

  30. [2025]

    arXiv:2503.01743 [cs.CL] doi:10.48550/arXiv.2503.01743

    Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs. arXiv:2503.01743 [cs.CL] doi:10.48550/arXiv.2503.01743

    doi arXiv

  31. [2026]

    arXiv:2601.22706 [cs.SE] doi:10.48550/arXiv.2601.22706

    RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories. arXiv:2601.22706 [cs.SE] doi:10.48550/arXiv.2601.22706

    doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.