Pith. sign in

REVIEW 2 major objections 5 minor 26 references

Can AI Write Compliant Code, and to What Extent? Evaluating SOC 2 Compliance of Claude Fable 5, Claude Opus 4.8, and Claude Opus 5 Across Four Use Cases

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read When no one mentions security, frontier models omit SOC 2 controls; one sentence naming the standard moved every case from 47–88% to 86–100%.

desk verdict Honest, transparent measurement of SOC 2 prompt effects, but the "naming the standard" mechanism is confounded with data sensitivity and needs an ablation before the causal story holds. read the letter →

arxiv 2608.07776 v1 pith:HHEFLX5C submitted 2026-08-07 cs.CR

Iccha Sethi , Herman Errico This is my paper · ORCID
classification cs.CR
keywords SOC2complianceAIcodegenerationpromptengineeringsecurecontrolsTrustServicesCriteriastaticanalysisTerraform
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether code generated by frontier language models carries the controls a SOC 2 program would check, and how much one sentence naming the standard changes the answer. In a factorial test of three models across four use cases, code written from a neutral task statement satisfied 47–88% of the rubric items, with the missing controls clustering where security is not part of the normal coding idiom, such as S3 hardening calls, retention, and MFA hooks. The same task plus a single directive sentence declaring the data sensitive and subject to SOC 2 moved every cell to 86–100%, worth 23 to 50 points, and removed every insecure construction the paper found. The paper argues this places the compliance knowledge inside the model while its default behavior leaves the controls outside the prompt, and it reports that model choice mattered least among the variables measured.

What carries the argument

The carrying mechanism is the SOC 2 directive: one appended sentence declaring the data sensitive and the artifact subject to SOC 2, held in context on first receipt, with no individual control named. Around it, the method builds 12-to-15-item binary rubrics per use case, each item mapped to a stated Trust Services Criterion and frozen before any generation, plus a three-rater scoring protocol — a regular-expression pattern scorer, a blinded semantic model grader, and human hand-verification — that separates what the prompt changes from what the measurement instruments miss. The rubrics and all prompts are published verbatim so readers can rescore the unchanged artifacts.

What would settle it

Run each of the four use cases at least three times per cell, neutral and directed, and compare the score distributions. If any directed draw lands inside the neutral range, or any neutral draw reaches the directed range, the claimed 23-to-50-point directive effect is not stable; a replication that finds neutral code matching directed conformance would also falsify the claim that the standard has to be named.

Watch

Extended reading notes

Core claim

The central claim is that compliance knowledge is latent, not automatic: when the prompt says nothing about security, these models produce code that runs, passes ordinary tests, and still fails a compliance review, because an omitted control leaves no wrong line of code behind. Conformance without prompting ranged from 47% to 88% and tracked whether a control is embedded in the code idiom — password hashing and storage_encrypted appear unasked, while four S3 hardening calls, retention, and MFA hooks do not. One sentence naming SOC 2 recovered most of the gap: directed conformance ran 86–100% in every cell, and three real vulnerabilities in the neutral code (a reachable debugger giving remote code execution, an unauthenticated download, and an endpoint returning all stored names and emails) all disappeared. A stable remainder survived the sentence, including MFA hooks, cookie flags, and account lifecycle items, which have to be named individually. The paper also establishes that a regular-expression compliance scorer is directionally biased, under-crediting well-engineered code in 21 of 27 disagreements with semantic grading and passing a defect whose non-compliant value was computed by a conditional, so plan-time or execution-time resolution and auditing of scanner passes are necessary.

Load-bearing premise

The load-bearing assumption is that one non-deterministic code generation per cell represents the model's behavior on that task; the paper states this is not defensible for fine differences and pre-commits to three generations per cell.

Editorial extensions

If this is right

  • Teams auditing model-generated code with literal pattern matching will under-credit well-engineered code and can pass real defects; configuration values must be resolved semantically (plan output or AST) and a scanner's passes checked as carefully as its failures.
  • A generic compliance instruction is a ceiling, not a cure: controls outside the model's conception of the task, such as MFA hooks, cookie flags, and account lifecycle, must be named explicitly in the instruction set.
  • Model selection is the wrong lever for compliance: same-generation models finished within one rubric item in all eight comparisons, and the largest cross-generation difference (25 points) was smaller than the 23-to-50-point directive effect.
  • Environment is a stronger lever than instruction: declarative Terraform, where security attributes sit in the resource schema, beat an imperative boto3 script doing comparable work by 30 points, which points to secure-by-default wrappers as the most direct remediation.
  • Null commission results should be read as 'none of the enumerated acts': expanding the search list turned a null into three real vulnerabilities in unchanged files, so reported commission counts are lower bounds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the prompt-shaped gap generalizes across vendors, the cheapest reliable fix is a persistent instruction file re-read each session rather than prompt text alone; the direct test the paper registered but did not run is placing such a file in the repository and rerunning the neutral prompts.
  • The idiom-account hypothesis — that conformance tracks whether the control is part of normal code — was generated from four data points after seeing them; a discriminating test would vary repository context and task wording to see whether defaults shift when surrounding code already carries compliance signal.
  • The application-layer defects (debugger, unauthenticated endpoints) sit beyond what plan-time policy gates can block, so even a well-gated infrastructure pipeline still needs semantic control verification for application code.
  • Because each cell is one non-deterministic generation, the exact percentages are single draws; the first replication with three generations per cell should report per-cell variance and paraphrase variants before fine rankings are taken seriously.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper reports a controlled, factorial evaluation of three Claude models (Fable 5, Opus 4.8, Opus 5) across four code-generation use cases (S3 CLI, authentication service, RDS Terraform, file-upload handler). Each cell was generated once from a neutral task statement and once from the same statement plus a single added sentence declaring the data sensitive and the artifact subject to SOC 2, producing 24 artifacts scored against binary rubrics mapped to Trust Services Criteria. The authors find unprompted conformance of 47-88% varying by whether controls are idiomatically part of the code, that the added sentence raises every cell to 86-100% (worth 23-50 points) and removes every insecure construction found, that model choice has little detectable effect, and that a regular-expression scorer is unreliable, under-crediting well-engineered code 21 of 27 times and passing a real defect expressed through a conditional. The paper is unusually transparent: rubrics frozen before generation, predictions registered for two use cases and for the later Opus 5 arm, all outputs published, and limitations stated at length, including the admission that one generation per cell does not support fine differences.

Significance. If the findings hold, the paper provides concrete evidence for a practically important phenomenon: frontier models can apply compliance controls when a generic security/compliance signal is present, but omit many of the same controls when it is absent, and the omission can coexist with real vulnerabilities. The main contribution is not a new benchmark but a carefully described measurement protocol with published rubrics, verbatim prompts, raw artifacts, and per-item hand-verified scores, which makes the study unusually easy to reproduce or rescore. The consistency of the prompt effect across all eight cells is a genuine strength, and the paper's treatment of the commission list as an instrument whose scope changes the answer is honest. The two most significant limitations are that the directive bundles two cues ('data sensitive' and 'subject to SOC 2'), so the specific causal role of naming the standard is not identified, and that every cell is a single non-deterministic generation, so the exact percentages and especially the cross-model comparisons sit at the resolution floor.

major comments (2)
  1. [§3.1, Abstract, §8] The directed condition is defined as the identical task 'plus one sentence declaring the data sensitive and the artifact subject to SOC 2' (§3.1), but the Abstract and §8 attribute the effect to 'one sentence naming the standard' and state that the default behavior omits controls 'unless the standard is named.' Because the single sentence varies two properties at once, the observed 23-50 point gains and removal of insecure constructions are compatible with a generic response to 'data sensitive' alone, with the SOC 2 label contributing little or nothing. The design never varies the two components, and the pre-committed paraphrase run does not separate them either. Please either add a sensitivity-only and a standard-only ablation, or recast the mechanistic claims so that they attribute the effect to the two-part directive rather than to naming the standard. This is load-bearing for the central explanatory claim in the Abstract and §8.
  2. [§4 (A4), §5.5, §7] All 24 cells are single non-deterministic generations, so the exact percentages, the stated size of the directive effect, and the cross-model conclusions in §5.5 rest on one draw per cell. The paper discloses this and pre-commits to three generations per cell, and the direction of the prompt effect is consistent across all eight cells, which buffers the main practical finding. However, the abstract's statement that 'model choice mattered least' and the specific claim that same-generation models are 'within one rubric item across all eight cells' are descriptions of single samples, not of model behavior; the paper itself correctly notes the resolution floor. A repetition run with three or more generations per cell is needed before the model comparison can be stated as more than a descriptive observation about these particular outputs. I would ask that the abstract's wording be brought in line with this limitation (the body already is).
minor comments (5)
  1. [§3.3] There is a spacing typo in 'F astAPI' that should read 'FastAPI'.
  2. [§5.1, Table 2] The text says declarative Terraform came in '30 points above' the boto3 script, but the subsequent AWS-defaults correction reduces the gap to 17 points; the sentence should acknowledge the corrected figure in the same paragraph to avoid an apparent inconsistency.
  3. [§5.5, Abstract] The abstract states that Opus 5 improved on the two weakest cases 'by less than the sentence was worth,' but for the upload case the improvement (25 points) equals the directive's effect for Opus 5 on that case, not less; the wording should be adjusted to 'at most as much as.'
  4. [§6] The phrase 'the −0.75 deficit therefore bounds total error from below' is confusing, since the deficit is only with respect to the scorer's negative errors and the paper explains that over-credits are unmeasured; consider rewording to 'the −0.75 mean signed error quantifies the known under-credit direction but does not bound the unknown over-credit rate.'
  5. [§7] The discussion of the Opus 5 arm's deviations from the original conditions is thorough, but the phrase 'its fresh-session requirement was met by isolated clean-context sessions in place of separate interactive ones' would benefit from one concrete example of how the sessions differed (e.g., new process versus new conversation), since the contamination risk is otherwise hard to assess.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the results are direct empirical measurements against frozen rubrics and pre-registered predictions, with no fitted parameter, no load-bearing self-citation, and no derivation that reduces to its own inputs.

full rationale

This paper performs no derivation in the sense that could reduce to its own inputs. The central quantities (unprompted conformance, directed-conformance gains, vulnerability counts, and rater disagreement) are direct measurements of 24 emitted artifacts against frozen rubrics and pre-registered predictions, with all artifacts published verbatim and every file:line citation resolving. No parameter is fitted to the outcome, and no 'prediction' is computed from a fitted value; indeed, two of the seven original pre-registered predictions were refuted, which cuts against the idea that the results are forced by construction. The self-referential elements the paper discloses—the post hoc expansion of the commission list after reading the corpus, the single-generation-per-cell design, and the idiom account being a post hoc reading of four data points—are threats to validity and interpretive limitations, not circular derivations. Expanding the list changed the measurement instrument while leaving the outputs untouched, and the authors explicitly label the idiom account as a hypothesis the study generated and did not test. The closest issue is that the directed condition bundles 'declaring the data sensitive' with 'subject to SOC 2' in one sentence, while the abstract and conclusion attribute the effect to 'one sentence naming the standard'; that is a construct-confounding design limitation requiring a sensitive-only ablation, not a case where the conclusion is equivalent to the input by definition or by self-citation. There are no load-bearing self-citations: the reference list contains no work by the authors, no uniqueness theorem is imported, and no ansatz is smuggled in via citation. The pattern-scorer critique is itself an empirical measurement against hand-verified semantic grading, including a false pass on a real defect, so it does not borrow the paper's conclusion to prove itself.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters because the study is a measurement, not a fit; percentages come from binary rubric counts. No invented entities and no new theoretical constructs. The five stated assumptions in Section 4 carry the interpretation, and the most fragile is A4, one generation per cell, which the paper flags as not defensible for fine differences.

assumptions (5)
  • domain assumption A1: Rubric items are valid proxies for their criteria.
    Each rubric item maps to a Trust Services Criterion, but satisfaction of a proxy does not prove conformance with the underlying control; Section 3.4.
  • domain assumption A2: Static inspection captures the compliance-relevant behavior of these artifacts.
    Nothing was executed and no runtime requests were sent, so runtime efficacy is unmeasured; Sections 3.5 and 7.
  • domain assumption A3: The commission list is complete enough that a null result would mean something.
    This assumption failed; the list was expanded after a null result and after reading the corpus, so commission counts are lower bounds; Sections 3.6 and 4.
  • domain assumption A4: One generation per cell represents the model's behavior on the task.
    Not defensible for fine differences, with one non-deterministic draw per cell and no variance estimate; Sections 4 and 7.
  • domain assumption A5: The neutral prompt is genuinely neutral.
    Prompts are published verbatim and contain no security language, but an empty working directory may read as a greenfield demonstration, which would understate real-repository behavior; Sections 4 and 7.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can AI Write Compliant Code, and to What Extent? Evaluating SOC 2 Compliance of Claude Fable 5, Claude Opus 4.8, and Claude Opus 5 Across Four Use Cases." pith.science (2026). https://pith.science/paper/HHEFLX5C

@misc{pith2026260807776,
  author       = {Pith},
  title        = {Pith review of: Can AI Write Compliant Code, and to What Extent? Evaluating SOC 2 Compliance of Claude Fable 5, Claude Opus 4.8, and Claude Opus 5 Across Four Use Cases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HHEFLX5C}},
  note         = {Machine review of arXiv:2608.07776}
}
read the original abstract

Software teams now delegate production code to language models, including code that provisions storage, handles credentials, and stores regulated data, so we asked whether a model applies the controls a SOC~2 program expects (encryption, restricted access, logging, retention) when nobody mentions security, and how much one sentence naming the standard changes the answer. We tested three frontier models (Claude Fable~5, Opus~4.8, and Opus~5) across four use cases (an S3 CLI, an authentication service, an RDS Terraform module, and a file-upload handler holding personal data), each generated once from a neutral task statement and once with a single SOC~2 sentence added, scoring all 24 outputs against binary rubrics mapped to specific Trust Services Criteria and hand-verifying every failure and flagged act. Unprompted conformance ran from 47\% to 88\% and tracked whether a control is part of how the code is normally written, so password hashing and \texttt{storage\_encrypted} appear unasked while S3 hardening calls, retention, and MFA hooks do not. The neutral prompt also shipped real vulnerabilities, including a reachable Werkzeug debugger allowing remote code execution, an unauthenticated download, and an endpoint returning every stored name and email, all scored clean by our first checklist, with a fourth defect passing because its value was computed by a conditional. One SOC~2 sentence moved every case to 86--100\%, worth 23 to 50 points, and removed every insecure construction, though controls outside the model's conception of the task survived it, including MFA hooks, cookie flags, and account lifecycle. Model choice mattered least, with same-generation models within one rubric item across all eight cells, and the pattern-matching scorer proved unreliable, disagreeing with semantic grading on 27 of 216 judgments and passing a real defect, so it needs replacing with semantic checks.

Figures

Figures reproduced from arXiv: 2608.07776 by the authors.

Figure 1
Figure 1. Conformance by use case and condition across three model arms. Neutral prompts in blue, directed in or￾ange, with the darkest bar of each shade being Opus 5. Within a shade the bars are close together, which is the model result; between shades they are far apart, which is the prompt result. models, 30 points above the boto3 script doing compara￾ble provisioning work. The variable that sorts the data is whether a con… view at source ↗
Figure 2
Figure 2. The directive buys most where unprompted de￾faults are weakest. Auth was already at 82% and had the least room, which is also where the ceiling of §5.3 appears. finished the directed round at 86%. Two items failed in every auth run and the directive did not recover either: an MFA hook, and Secure/HttpOnly/SameSite cookie flags. Both services issued bearer tokens and set no cook￾ies, which arguably makes the cookie i… view at source ↗
Figure 3
Figure 3. Rater totals. The axis is broken: it starts at 145, marked at the origin, so the bars show differences between raters and cannot be read as proportions of 216. The blinded grader lands close to the human total and both sit above the pattern scorer. generation from a non-deterministic system, so we cannot estimate within-cell variance, differences of one to two rubric items are indistinguishable from sampling noise, … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 19 canonical work pages

  1. [1]

    Asleep at the Keyboard? Assessing the Secu- rity of GitHub Copilot’s Code Contributions,

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the Keyboard? Assessing the Secu- rity of GitHub Copilot’s Code Contributions,” inIEEE Symposium on Security and Privacy (S&P), 2022

    work page 2022

  2. [2]

    Do Users Write More Insecure Code with AI Assistants?

    N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do Users Write More Insecure Code with AI Assistants?” in ACM SIGSAC Conference on Computer and Communi- cations Security (CCS), 2023

    work page 2023

  3. [3]

    Lost at C: A User Study on the Secu- rity Implications of Large Language Model Code Assis- tants,

    G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. Dolan-Gavitt, “Lost at C: A User Study on the Secu- rity Implications of Large Language Model Code Assis- tants,” inUSENIX Security Symposium, 2023

    work page 2023

  4. [4]

    LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations,

    C. Tony, M. Mutas, N. E. Díaz Ferreyra, and R. Scan- dariato, “LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations,” inIEEE/ACM In- ternational Conference on Mining Software Repositories (MSR), 2023

    work page 2023

  5. [5]

    SecurityEval Dataset: Mining Vulnerability Examples to Evaluate Machine Learning-Based Code Generation Techniques,

    M. L. Siddiq and J. C. S. Santos, “SecurityEval Dataset: Mining Vulnerability Examples to Evaluate Machine Learning-Based Code Generation Techniques,” inMSR4P&S Workshop (ESEC/FSE), 2022

    work page 2022

  6. [6]

    SALLM: Security Assessment of Generated Code,

    M. L. Siddiq, J. C. S. Santos, S. Devareddy, and A. Muller, “SALLM: Security Assessment of Generated Code,” in39th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW, ASYDE), 2024, arXiv:2311.00889

    arXiv 2024

  7. [7]

    CodeLMSec Benchmark: Systematically Eval- uating and Finding Security Vulnerabilities in Black-Box Code Language Models,

    H. Hajipour, K. Hassler, T. Holz, L. Schönherr, and M. Fritz, “CodeLMSec Benchmark: Systematically Eval- uating and Finding Security Vulnerabilities in Black-Box Code Language Models,” inIEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2024

    work page 2024

  8. [8]

    Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models,

    M. Bhatt et al., “Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models,” arXiv:2312.04724, 2023

    arXiv 2023

Show all 26 references
  1. [9]

    CW- Eval: Outcome-Driven Evaluation on Functionality and Security of LLM Code Generation,

    J. Peng, L. Cui, K. Huang, J. Yang, and B. Ray, “CW- Eval: Outcome-Driven Evaluation on Functionality and Security of LLM Code Generation,” inLLM4Code Work- shop (ICSE), 2025, arXiv:2501.08200. 11

    2025 arXiv

  2. [10]

    Prompting Techniques for Secure Code Generation: A Systematic Investigation,

    C. Tony, N. E. Díaz Ferreyra, M. Mutas, S. Dhiff, and R. Scandariato, “Prompting Techniques for Secure Code Generation: A Systematic Investigation,”ACM Transactions on Software Engineering and Methodology (TOSEM), 2025, arXiv:2407.07064

    2025 arXiv

  3. [11]

    Benchmarking Prompt Engineering Techniques for Se- cureCodeGenerationwithGPTModels,

    M. Bruni, F. Gabrielli, M. Ghafari, and M. Kropp, “Benchmarking Prompt Engineering Techniques for Se- cureCodeGenerationwithGPTModels,” inIEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE),2025, arXiv:2502.06039

    2025 arXiv

  4. [12]

    How Secure is Code Generated by ChatGPT?

    R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara, “How Secure is Code Generated by ChatGPT?” inIEEE International Conference on Systems, Man, and Cyber- netics (SMC), 2023

    2023

  5. [13]

    Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena,

    L. Zheng et al., “Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena,” inAdvances in Neural In- formation Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023

    2023

  6. [14]

    G-Eval: NLG Evaluation using GPT-4 with Better Hu- man Alignment,

    Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-Eval: NLG Evaluation using GPT-4 with Better Hu- man Alignment,” inConference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  7. [15]

    Large Language Models are not Fair Evaluators,

    P. Wang et al., “Large Language Models are not Fair Evaluators,” inAnnual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 9440–9450, arXiv:2305.17926

    2024 arXiv

  8. [16]

    How Secure is Secure Code Generation? Ad- versarial Prompts Put LLM Defenses to the Test,

    M. Tessa, I. E. Olatunji, A. War, J. Klein, and T. F. Bis- syandé, “How Secure is Secure Code Generation? Ad- versarial Prompts Put LLM Defenses to the Test,” arXiv:2601.07084, 2026

    2026

  9. [17]

    Why Don’t Software Developers Use Static Analysis Tools to Find Bugs?

    B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, “Why Don’t Software Developers Use Static Analysis Tools to Find Bugs?” inInternational Conference on Software Engineering (ICSE), 2013, pp. 672–681

    2013

  10. [18]

    A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World,

    A. Bessey et al., “A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World,” Communications of the ACM, vol. 53, no. 2, pp. 66–75, 2010

    2010

  11. [19]

    A Coefficient of Agreement for Nominal Scales,

    J. Cohen, “A Coefficient of Agreement for Nominal Scales,”Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960

    1960

  12. [20]

    Checkov: policy-as- code static analysis for infrastructure as code,

    Bridgecrew / Palo Alto Networks, “Checkov: policy-as- code static analysis for infrastructure as code,”https: //www.checkov.io/

  13. [21]

    tfsec: security scanner for Terraform code,

    Aqua Security, “tfsec: security scanner for Terraform code,”https://github.com/aquasecurity/tfsec

  14. [22]

    OPA / Rego policy en- gine,

    Open Policy Agent (CNCF), “OPA / Rego policy en- gine,”https://www.openpolicyagent.org/

  15. [23]

    Large Language Models for Code: Security Hardening and Adversarial Testing,

    J. He and M. Vechev, “Large Language Models for Code: Security Hardening and Adversarial Testing,” inACM SIGSAC Conference on Computer and Communications Security (CCS), 2023

    2023

  16. [24]

    In- struction Tuning for Secure Code Generation,

    J. He, M. Vero, G. Krasnopolska, and M. Vechev, “In- struction Tuning for Secure Code Generation,” inInter- national Conference on Machine Learning (ICML), 2024, arXiv:2402.09497

    2024 arXiv

  17. [25]

    Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study,

    Y. Fu, P. Liang, A. Tahir, Z. Li, M. Shahin, J. Yu, and J. Chen, “Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study,” ACM Trans. on Software Engineering and Methodology (TOSEM), 2025, arXiv:2310.02059

    2025 arXiv

  18. [26]

    Lessons from Building Static Analysis Tools at Google,

    C. Sadowski, E. Aftandilian, A. Eagle, L. Miller-Cushon, and C. Jaspan, “Lessons from Building Static Analysis Tools at Google,”Communications of the ACM, vol. 61, no. 4, pp. 58–66, 2018. 12

    2018

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.