Pith. sign in

REVIEW 3 major objections 5 minor 52 references

The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Removing a single syntactic constituent from a code-generation prompt—a constraint, guard, condition, or binding phrase—can substantially raise the chance that an open LLM produces vulnerable code.

desk verdict Novel and worth reviewing, but the 'removal increases risk' claims compare deletions to other deletions, not to the intact prompt; re-baseline before trusting the causal language. read the letter →

arxiv 2607.15937 v1 pith:L47VQKBD submitted 2026-07-17 cs.CR cs.SE

Matteo Cicalese , Antonio Della Porta , Stefano Lambiase , Emanuele Iannone , Torge Hinrichs , Riccardo Scandariato , Fabio Palomba This is my paper · ORCID
classification cs.CRcs.SE
keywords promptsyntaxsecurecodegenerationsyntacticconstituentsconstituencyparsingopenLLMsvulnerabilitypredictionpositionalbiasstaticanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that prompt syntax is a security control surface: how a code-generation request is grammatically put together, not just what it asks for, changes how often the resulting code is vulnerable. It tests this by deleting one syntactic constituent at a time from security-relevant prompts, generating code with three open LLMs in C, Java, and Python, and statistically comparing vulnerability rates. The core finding is that both the type of the deleted constituent and its position in the prompt matter: deleting interrogative/relative binding phrases, action verb phrases, conditional guards, and qualifier adjectives raises vulnerability risk, with Java most sensitive at the opening sentence and Python increasingly sensitive toward later sentences. If the finding holds, prompt phrasing becomes a concrete, low-cost lever for reducing vulnerabilities in LLM-assisted development, and syntactic detail deserves as much attention as high-level prompting strategy.

What carries the argument

The operational mechanism is a 'prompt permutation': a prompt with exactly one syntactic constituent removed. Each deletion is characterized by three features—constituent type (the grammatical label of the removed span), granularity (minimal, chunk, or clause), and sentence index (its position in the prompt). The paper builds these permutations from constituency parse trees, uses them to generate code, labels vulnerability with static analysis, and then tests each feature group against the rest of the edited set using omnibus chi-square tests followed by exact pairwise comparisons with risk ratios. The constituent type and sentence-index features carry most of the explanatory weight, while g

What would settle it

Rerun the same deletion study with the original unedited prompts as the only control and with multiple code generations per prompt; if headline increases such as the Phi-4/Java first-sentence deletion (risk ratio 2.31) disappear or reverse when compared to the unedited baseline, the claim that the removed syntax itself drives vulnerability risk loses support.

Watch

Extended reading notes

Core claim

The paper's central claim is that fine-grained syntax in prompts systematically shifts the security of LLM-generated code, in a way that is predictable from the type and position of the removed constituent. Its experiments show that across three open models, deleting constituents associated with entity/constraint binding (relative/interrogative phrases), action semantics (verb phrases), conditional guards (subordinate clauses), and normative qualifiers (adjectival phrases) increases vulnerability rates, with the largest effects in Python (risk ratios around 1.7–2.2, and tail-position rates approaching 100%) and a Java-specific effect at the first sentence. The direction of these effects is c

Load-bearing premise

The causal claim rests on treating the other edited prompts as a valid control group: each deletion's effect is measured against the remainder of the edited set rather than against the original unedited prompt, and a single round of generation per prompt is taken as representative.

Editorial extensions

If this is right

  • Prompt syntax becomes a concrete control surface: preserving WH/relative binding phrases, action verb phrases, conditional guards, and qualifier adjectives should reduce the chance of insecure output.
  • Position is a first-class variable: Java prompts concentrate risk in the opening sentence; Python prompts grow riskier toward the tail, so where a constraint sits matters as much as whether it is present.
  • The effects are stable in direction across three differently sized open models, meaning model choice alone will not erase syntactic sensitivity.
  • Even minimal edits can be dangerous: removing one small constituent is enough to raise vulnerability rates, so casual shortening or paraphrasing of prompts carries security risk.
  • Syntax-aware prompt review—checking that key binding and guard phrases are present and well-placed—could be built into pre-generation tooling as a complement to post-hoc vulnerability scanning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because only open models were tested, whether proprietary models show the same directional sensitivity remains open; the paper's cross-model consistency suggests it may, but that is an extrapolation.
  • Inference: the paper's 'anchor vs. tail' reading could be directly probed by moving a critical constraint from the first sentence to the last while keeping wording identical; the positional-gradient hypothesis predicts that vulnerability risk follows the slot, not the sentence's content.
  • Inference: the weak C-language results may reflect a floor effect—C's vulnerability rate in this corpus is near 1%, leaving little room to detect increases—so C-specific vulnerability-rich prompts would be needed before concluding C syntax is insensitive.
  • Inference: a prompt linter based on constituent-deletion patterns is a natural extension; its usefulness would be testable by rewriting flagged prompts to restore missing WHNP/WHADVP/SBAR/VP elements and measuring whether vulnerability rates fall.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies how fine-grained syntactic constituents of code-generation prompts affect the security of code produced by open-weight LLMs. Starting from the 150 LLMSecEval prompts, the authors parse each prompt with a constituency parser, generate 12,960 prompt variants by deleting a single syntactic constituent, and run Qwen 2.5 32B, Phi-4 14.7B, and Athene-V2 72B to produce code in C, Java, and Python. CodeQL labels snippets as vulnerable. The authors analyze vulnerability rates by constituent type, sentence index, and granularity, using omnibus chi-square tests, Barnard exact tests, and risk ratios, and report that position and constituent type—especially WHNP/WHADVP binding constructions—consistently affect vulnerability incidence, with Java risk concentrated at the first sentence and Python risk increasing toward later positions. They conclude that prompt syntax is a security control surface and give guidance to preserve security-relevant constituents.

Significance. If the causal claims are supported, this is a useful contribution: it moves beyond coarse prompting strategies, targets open models, uses a controlled parser-driven perturbation design, and ships a replication package. The confirmatory subsample with multiple generations per prompt is a genuine strength, and the cross-model consistency (at least for Java/Python) is suggestive. However, the main inferential reference class is the pooled set of other prompt deletions rather than the intact prompt, so the headline 'removal increases risk' claims are not established as stated. The paper is worth publishing after re-baselining or reframing.

major comments (3)
  1. [III-D3, Tables II-IV] The inferential step compares each feature level with 'the remainder of the dataset' of prompt permutations, not with the unperturbed baseline prompts. The risk ratios in Table IV are therefore relative to the pooled other deletion classes, not to the intact prompt. A concrete instance: Phi-4/Java Sentence Index 0 has a vulnerability rate of 8.55% (Table III), below the 9.17% baseline for Phi-4/Java (Table II), yet Table IV reports RR=2.31 (p=0.001) and the text (Section IV-B1) calls this a significant 'increase'. The increase is relative to a small remainder subset, not to the original prompt. Because the abstract and conclusions assert that removing constituents makes code 'more likely' to be insecure, this reference class is load-bearing. Please re-run the localization tests against the original baseline, or restrict the claims to 'some deletions are riskier than other deletions'. The
  2. [IV-C and IV-B (C language)] All C-language omnibus tests are reported as N/A (Table IV), and the baseline C vulnerability rate is below 1.5% with only 14-21 vulnerable permutations per model (Table II). Nevertheless, Section IV-C states that the three critical patterns 'hold across all tested LLMs for each language' and that the criticality ordering holds across C, Java, and Python. The C support consists of descriptive differences of at most a few percentage points with no inferential test; this is insufficient to claim cross-language consistency. Either provide a test with sufficient power for C or soften the claim to Java/Python.
  3. [VI (Internal validity) and III-D3] The significance tests treat every generated snippet as an independent Bernoulli observation. However, the 4,320 permutations per model-language are generated from only 150 base prompts, so permutations derived from the same prompt are correlated; in addition, the main results use one sample per prompt. The confirmatory subsample with three samples per prompt checks generation variance but does not remove the clustering of permutations within base prompts. As a result, the reported p-values are likely anti-conservative. Please use a cluster-robust or mixed-effects analysis, or at least acknowledge this limitation explicitly in the inferential interpretation.
minor comments (5)
  1. [IV-A1] The text says 'Java had the highest failure rate, of 86.81%', but Table II shows Java success rates around 80-93%; this is likely a typo for 'success rate of 86.81%' or 'failure rate of 13.19%'.
  2. [VI (Conclusion validity)] The 'quantile-based sufficiency filter' is mentioned but never defined. Specify the quantile threshold and how cells were excluded.
  3. [Table III] The Granularity column for Phi-4/Java is 'N/A' without explanation. Clarify whether no granularity feature met the reporting threshold or whether the data are missing.
  4. [III-D1 vs III-D3] Section III-D1 says rates are compared with the 'baseline vulnerability rate of the corresponding permutation set', while III-D3 says comparisons are against the 'remainder of the dataset'. These are different references; align the terminology with the actual procedure.
  5. [Table IV] Athene/Python Sentence Index 2 reports p=92.016e-37; use standard scientific notation (9.20e-36) for readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: empirical perturbation study with external CodeQL labeling; the central inference is not equivalent to its inputs.

full rationale

The paper's derivation chain is empirical, not algebraic: prompts from the public LLMSecEval dataset are parsed with crf-con-en, each perturbation removes one Penn Treebank constituent, three open LLMs generate code, and CodeQL labels vulnerabilities. The feature labels (Constituent Type, Granularity, Sentence Index) come from an external parser and the outcome labels from an external static analyzer, so no quantity used in the conclusion is fitted from the outcome or defined in terms of the conclusion. The self-citations to LLMSecEval and prompting work by the same group are real but not load-bearing: the dataset is a public artifact and the security labels are independent of the dataset itself. The main inferential caveat is in Section III-D3, where each feature level is compared with "the remainder of the dataset" using Barnard's exact test rather than with the unperturbed original prompts; this weakens the causal phrasing ("removal increases vulnerability") but is a reference-class issue, not a circular reduction, because the comparison is still against observed CodeQL rates. The paper also acknowledges parser and static-analysis threats in Section VI. No step reduces by construction to its own input, so the circularity score is low.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

No new entities or fitted physical parameters are introduced. The claim rests on domain assumptions about the parser, the vulnerability oracle, controlled deletion, sampling stability, statistical independence, and dataset representativeness, plus researcher-chosen analytic thresholds.

free parameters (3)
  • FDR significance threshold
    Benjamini-Hochberg control is mentioned but the q-threshold is not reported; a standard 0.05 is assumed.
  • Quantile-based sufficiency filter
    Mentioned in the threats-to-validity section but no quantile or minimum cell size is specified; this is an unstated analytic choice that can change which configurations are tested.
  • RR<1 exclusion rule
    Statistically significant configurations with RR<1 are excluded from discussion as 'linguistically non-impactful,' which is a one-sided interpretive choice shaping the reported risk picture.
assumptions (6)
  • domain assumption Constituency parse trees from crf-con-en with Penn Treebank II categories provide a valid decomposition of prompts for causal analysis.
    Section III-A and Definition 1: the entire intervention set is built on this parser's tree structure; if parser output does not align with model-relevant linguistic structure, the constituent-type results are artifacts.
  • domain assumption CodeQL default query suites produce trustworthy vulnerability labels for LLM-generated snippets in C, Java, and Python.
    Section III-C: all outcome labels come from CodeQL; false negatives and false positives propagate into every rate and risk ratio.
  • domain assumption Removal of exactly one constituent is a controlled intervention, with changes in vulnerability attributable to the removed constituent and not to changed length, fluency, or topic drift.
    Section III-A and Figure 2; the causal wording of RQ1/RQ2 requires this.
  • domain assumption LLM generation is stable enough for one sample per prompt to represent the model's behavior.
    Section III-B and Section VI: the main results rely on one generation per prompt; deterministic decoding is claimed but sampling parameters are not fully specified. A confirmatory subsample uses three samples for a subset.
  • standard math Statistical tests treat snippets sharing prompt roots and model outputs as conditionally independent observations.
    Section III-D3: repeated prompts and models are treated as independent in omnibus and exact tests, which can inflate significance.
  • domain assumption LLMSecEval's 150 prompts are representative of realistic security-relevant developer requests.
    Section III-A: prompts are drawn from MITRE Top 25 scenarios; external validity and the general 'prompt syntax matters' conclusion depend on this representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs." pith.science (2026). https://pith.science/paper/L47VQKBD

@misc{pith2026260715937,
  author       = {Pith},
  title        = {Pith review of: The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L47VQKBD}},
  note         = {Machine review of arXiv:2607.15937}
}
read the original abstract

Large Language Models (LLMs) are increasingly used for source code generation despite their outputs often exhibiting security vulnerabilities. Prior work shows that prompt engineering can mitigate such risks, yet (1) they focused on high-level prompting strategies, neglecting recent evidence that fine-grained syntactic variations can substantially alter model behavior; and (2) predominantly evaluate proprietary LLMs, limiting the applicability of their findings in industrial settings where self-hosted, open models are preferred for privacy, compliance, and deployment control. In this paper, we study how fine-grained syntactic constituents of prompts influence the security of open LLM-generated code. Using a parser-driven approach, we systematically generate syntactic variants of security-relevant code generation prompts and evaluate their impact on code security across multiple open LLMs and programming languages. Our results show that specific syntactic elements, such as constraints, guards, conditions, and concept bindings, and their position within the prompt consistently affect the likelihood of generating insecure code. These findings identify prompt syntax as a concrete security control surface and provide actionable guidance for reducing vulnerability risk in LLM-assisted development.

Figures

Figures reproduced from arXiv: 2607.15937 by the authors.

Figure 1
Figure 1. Overview of the Experimental Pipeline. In particular, we employ crf-con-en, an English constituency parser [38] that represents each sentence as a hierarchical phrase-structure tree, grouping words into grammatical con￾stituents and capturing how these constituents combine to form the full sentence. This parser, which has been widely used in prior studies on prompt and text structure analysis [37], [39], [40], follo… view at source ↗
Figure 2
Figure 2. Example of the Applied Permutation Process. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 5 linked inside Pith

  1. [1]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  2. [2]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,”ACM Transactions on Software Engineering and Methodology, 2024

    2024

  3. [3]

    Fairness set and forgotten: Mining fairness toolkit usage in open-source machine learning projects,

    A. Cannavale, G. V oria, A. Scognamiglio, G. Giordano, G. Catolino, and F. Palomba, “Fairness set and forgotten: Mining fairness toolkit usage in open-source machine learning projects,”Information and Software Technology, p. 107957, 2025

    2025

  4. [4]

    Contextual fairness-aware practices in ml: A cost-effective empirical evaluation,

    A. Parziale, G. V oria, G. Giordano, G. Catolino, G. Robles, and F. Palomba, “Contextual fairness-aware practices in ml: A cost-effective empirical evaluation,” in2025 IEEE International Conference on Soft- ware Analysis, Evolution and Reengineering-Companion (SANER-C). IEEE, 2025, pp. 1–8

    2025

  5. [5]

    Fairness on a budget, across the board: A cost-effective eval- uation of fairness-aware practices across contexts, tasks, and sensitive attributes,

    ——, “Fairness on a budget, across the board: A cost-effective eval- uation of fairness-aware practices across contexts, tasks, and sensitive attributes,”Information and Software Technology, p. 107858, 2025

    2025

  6. [6]

    Fair and square? evaluating fairness of llm-generated synthetic datasets,

    G. V oria, B. Scala, L. Todisco, C. Venditto, G. Giordano, G. Catolino, and F. Palomba, “Fair and square? evaluating fairness of llm-generated synthetic datasets,”Information and Software Technology, p. 107980, 2025

    2025

  7. [7]

    Human-written vs. ai- generated code: A large-scale study of defects, vulnerabilities, and complexity,

    D. Cotroneo, C. Improta, and P. Liguori, “Human-written vs. ai- generated code: A large-scale study of defects, vulnerabilities, and complexity,” in2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2025, pp. 252–263

    2025

  8. [8]

    Security vulnerabilities in ai-generated code: A large-scale analysis of public github repositories,

    M. Schreiber and P. Tippe, “Security vulnerabilities in ai-generated code: A large-scale analysis of public github repositories,” inInternational Conference on Information and Communications Security. Springer, 2025, pp. 153–172

    2025

Show all 52 references
  1. [9]

    Security degradation in iterative ai code generation: A systematic analysis of the paradox,

    S. Shukla, H. Joshi, and R. Syed, “Security degradation in iterative ai code generation: A systematic analysis of the paradox,” in2025 IEEE International Symposium on Technology and Society (ISTAS). IEEE, 2025, pp. 1–8

    2025

  2. [10]

    Just another copy and paste? comparing the security vulnerabilities of chatgpt generated code and stackoverflow answers,

    S. Hamer, M. d’Amorim, and L. Williams, “Just another copy and paste? comparing the security vulnerabilities of chatgpt generated code and stackoverflow answers,” in2024 IEEE Security and Privacy Workshops (SPW). IEEE, 2024, pp. 87–94

    2024

  3. [11]

    Do users write more insecure code with ai assistants?

    N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with ai assistants?” inProceedings of the 2023 ACM SIGSAC conference on computer and communications security, 2023, pp. 2785–2799

    2023

  4. [12]

    Asleep at the keyboard? assessing the security of github copilot’s code con- tributions,

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code con- tributions,”Communications of the ACM, vol. 68, no. 2, pp. 96–105, 2025

    2025

  5. [13]

    When code smells meet ml: on the lifecycle of ml-specific code smells in ml-enabled systems,

    G. Recupito, G. Giordano, F. Ferrucci, D. Di Nucci, and F. Palomba, “When code smells meet ml: on the lifecycle of ml-specific code smells in ml-enabled systems,”Empirical Software Engineering, vol. 30, no. 5, p. 139, 2025

    2025

  6. [14]

    Into the ml-universe: An improved classification and characterization of machine-learning projects,

    V . De Martino, G. Recupito, G. Giordano, F. Ferrucci, D. Di Nucci, and F. Palomba, “Into the ml-universe: An improved classification and characterization of machine-learning projects,”Journal of Systems and Software, vol. 230, p. 112471, 2025

    2025

  7. [15]

    An evidence- based study on the relationship of software engineering practices on code smells in python ml projects,

    G. Giordano, A. Della Porta, F. Ferrucci, and F. Palomba, “An evidence- based study on the relationship of software engineering practices on code smells in python ml projects,” inEuromicro Conference on Software Engineering and Advanced Applications. Springer, 2025, pp. 105–120

    2025

  8. [16]

    Under- standing developer practices and code smells diffusion in ai-enabled software: A preliminary study

    G. Giordano, G. Annunziata, A. De Lucia, F. Palombaet al., “Under- standing developer practices and code smells diffusion in ai-enabled software: A preliminary study.” inIWSM-Mensura, 2023

    2023

  9. [17]

    Benchmarking prompt engineering techniques for secure code generation with gpt models,

    M. Bruni, F. Gabrielli, M. Ghafari, and M. Kropp, “Benchmarking prompt engineering techniques for secure code generation with gpt models,” in2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge). IEEE, 2025, pp. 93–103

    2025

  10. [18]

    Llmseceval: A dataset of natural language prompts for security evaluations,

    C. Tony, M. Mutas, N. E. D. Ferreyra, and R. Scandariato, “Llmseceval: A dataset of natural language prompts for security evaluations,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 2023, pp. 588–592

    2023

  11. [19]

    Prompting techniques for secure code generation: A systematic investi- gation,

    C. Tony, N. E. D ´ıaz Ferreyra, M. Mutas, S. Dhif, and R. Scandariato, “Prompting techniques for secure code generation: A systematic investi- gation,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 8, pp. 1–53, 2025

    2025

  12. [20]

    Retrieve, refine, or both? using task-specific guidelines for secure python code generation,

    C. Tony, E. Iannone, and R. Scandariato, “Retrieve, refine, or both? using task-specific guidelines for secure python code generation,” in2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2025, pp. 368–379

    2025

  13. [21]

    Prompt variability effects on llm code generation,

    A. Paleyes, R. Sendyka, D. Robinson, C. Cabrera, and N. D. Lawrence, “Prompt variability effects on llm code generation,”arXiv preprint arXiv:2506.10204, 2025

    2025

  14. [22]

    Nlperturbator: Studying the robustness of code llms to natural language variations,

    J. Chen, L. Zhenhao, H. Xing, and X. Xin, “Nlperturbator: Studying the robustness of code llms to natural language variations,”ACM Transactions on Software Engineering and Methodology, 2024

    2024

  15. [23]

    Toward measuring prompt quality: A preliminary investigation on prompt smells,

    A. Della Porta, G. V oria, A. Abbate, R. Sulipano, S. Lambiase, G. Catolino, and F. Palomba, “Toward measuring prompt quality: A preliminary investigation on prompt smells,” in2026 IEEE Interna- tional Conference on Software Analysis, Evolution and Reengineering- Companion (SA...

    2026

  16. [24]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,

    I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar, “Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,”arXiv preprint arXiv:2410.05229, 2024

    2024 arXiv

  17. [25]

    Using large language models to support software engineering documentation in waterfall life cycles: Are we there yet?

    A. Della Porta, V . De Martino, G. Recupito, C. Iemmino, G. Catolino, D. Di Nucci, F. Palombaet al., “Using large language models to support software engineering documentation in waterfall life cycles: Are we there yet?”Ital-IA, pp. 42–47, 2024

    2024

  18. [26]

    Does the grammatical structure of prompts influence the responses of generative artificial intelligence? an exploratory analysis in spanish,

    R. Viveros-Mu ˜noz, J. Carrasco-S ´aez, C. Contreras-Saavedra, S. San- Mart´ın-Quiroga, and C. E. Contreras-Saavedra, “Does the grammatical structure of prompts influence the responses of generative artificial intelligence? an exploratory analysis in spanish,”Applied Sciences,...

    2025

  19. [27]

    Security and privacy challenges of large language models: A survey,

    B. C. Das, M. H. Amini, and Y . Wu, “Security and privacy challenges of large language models: A survey,”ACM Computing Surveys, vol. 57, no. 6, pp. 1–39, 2025

    2025

  20. [28]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskillet al., “On the opportunities and risks of foundation models,”arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  21. [29]

    Cicalese, A

    M. Cicalese, A. Della Porta, S. Lambiase, E. Iannone, T. Hinrichs, R. Scandariato, and F. Palomba, https://doi.org/10.6084/m9.figshare. 32707911

    doi

  22. [30]

    Connecting large language models with evolution- ary algorithms yields powerful prompt optimizers,

    Q. Guo, R. Wang, J. Guo, B. Li, K. Song, X. Tan, G. Liu, J. Bian, and Y . Yang, “Connecting large language models with evolution- ary algorithms yields powerful prompt optimizers,”arXiv preprint arXiv:2309.08532, 2023

    2023 arXiv

  23. [31]

    Sallm: Security assessment of generated code,

    M. L. Siddiq, J. C. da Silva Santos, S. Devareddy, and A. Muller, “Sallm: Security assessment of generated code,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops, 2024, pp. 54–65

    2024

  24. [32]

    Devgpt: Studying developer-chatgpt conversations,

    T. Xiao, C. Treude, H. Hata, and K. Matsumoto, “Devgpt: Studying developer-chatgpt conversations,” inProceedings of the 21st interna- tional conference on mining software repositories, 2024, pp. 227–230

    2024

  25. [33]

    Do prompt patterns affect code quality? a first empirical assessment of chatgpt-generated code,

    A. Della Porta, S. Lambiase, and F. Palomba, “Do prompt patterns affect code quality? a first empirical assessment of chatgpt-generated code,” inProceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, 2025, pp. 181–192

    2025

  26. [34]

    Unlocking code simplicity: The role of prompt patterns in managing llm code complexity,

    A. Della Porta, G. Recupito, S. Lambiase, D. Di Nucci, and F. Palomba, “Unlocking code simplicity: The role of prompt patterns in managing llm code complexity,” in2025 IEEE International Conference on Software Analysis, Evolution and Reengineering-Companion (SANER-C). IEEE, 20...

    2025

  27. [35]

    Selective prompt anchoring for code generation,

    Y . Tian and T. Zhang, “Selective prompt anchoring for code generation,” arXiv preprint arXiv:2408.09121, 2024

    2024 arXiv

  28. [36]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. C. Ohlsson, B. Regnell, A. Wessl´en et al.,Experimentation in software engineering. Springer, 2012, vol. 236

    2012

  29. [38]

    Fast and accurate neural crf constituency parsing,

    Y . Zhang, H. Zhou, and Z. Li, “Fast and accurate neural crf constituency parsing,” inProceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI), 2020, pp. 4044–4051. [Online]. Available: https://www.ijcai.org/proceedings/2020/560

    2020

  30. [39]

    Grips: Gradient-free, edit-based instruction search for prompting large language models,

    A. Prasad, P. Hase, X. Zhou, and M. Bansal, “Grips: Gradient-free, edit-based instruction search for prompting large language models,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp. 3845–3864

    2023

  31. [40]

    Hierarchical fine-grained state-aware graph attention network for dialogue state tracking,

    H. Liao, Y . Chen, D. Chen, J. Xu, J. Zhong, and C. Dong, “Hierarchical fine-grained state-aware graph attention network for dialogue state tracking,”The Journal of Supercomputing, vol. 81, no. 5, p. 671, 2025

    2025

  32. [41]

    Available: https://surdeanu.cs.arizona.edu/mihai/teaching/ ista555-fall13/readings/PennTreebankConstituents.html

    [Online]. Available: https://surdeanu.cs.arizona.edu/mihai/teaching/ ista555-fall13/readings/PennTreebankConstituents.html

  33. [42]

    Building a large annotated corpus of English: The Penn Treebank,

    M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a large annotated corpus of English: The Penn Treebank,”Computational Linguistics, vol. 19, no. 2, pp. 313–330, 1993

    1993

  34. [43]

    An answer recommendation algo- rithm based on semantic fusion heterogeneous information network,

    Y . Wu, D. Liu, Q. Zhou, and H. Yin, “An answer recommendation algo- rithm based on semantic fusion heterogeneous information network,” in 2022 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI). IEEE, 2022, pp. 63–67

    2022

  35. [44]

    Text semantic representation based on knowledge graph correction,

    Y . Wu, H. Yin, D. Liu, and Q. Zhou, “Text semantic representation based on knowledge graph correction,” in2022 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI). IEEE, 2022, pp. 404–408

    2022

  36. [45]

    Knowledge graph- based hierarchical text semantic representation,

    Y . Wu, X. Pan, J. Li, S. Dou, J. Dong, and D. Wei, “Knowledge graph- based hierarchical text semantic representation,”International journal of intelligent systems, vol. 2024, no. 1, p. 5583270, 2024

    2024

  37. [46]

    Codelm- sec benchmark: Systematically evaluating and finding security vulnera- bilities in black-box code language models,

    H. Hajipour, K. Hassler, T. Holz, L. Sch ¨onherr, and M. Fritz, “Codelm- sec benchmark: Systematically evaluating and finding security vulnera- bilities in black-box code language models,” in2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2024, p...

    2024

  38. [47]

    An exploratory study on fine-tuning large language models for secure code generation,

    J. Li, F. Rabbi, C. Cheng, A. Sangalay, Y . Tian, and J. Yang, “An exploratory study on fine-tuning large language models for secure code generation,”arXiv preprint arXiv:2408.09078, 2024

    2024

  39. [48]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the association for computational linguistics, vol. 12, pp. 157–173, 2024

    2024

  40. [49]

    Empirical analysis of security vulnerabilities in python packages,

    M. Alfadel, D. E. Costa, and E. Shihab, “Empirical analysis of security vulnerabilities in python packages,”Empirical Software Engineering, vol. 28, no. 3, p. 59, 2023. [Online]. Available: https://dl.acm.org/doi/10.1007/s10664-022-10278-4

    2023 doi

  41. [50]

    Veracode uncovers the top security issues facing specific programming languages,

    C. Cardoza, “Veracode uncovers the top security issues facing specific programming languages,” SD Times, December 2020, accessed 2025- 10-01

    2020

  42. [51]

    Shift to memory-safe languages gains momen- tum,

    R. Lemos, “Shift to memory-safe languages gains momen- tum,” Dark Reading, December 2022, accessed 2025-10-01. [Online]. Available: https://www.darkreading.com/application-security/ shift-memory-safe-languages-gains-momentum

    2022

  43. [52]

    Software vulnerability analysis across programming language and program representation landscapes: A survey,

    Z. Qian, F. Zhong, Q. Hu, Y . Jiang, J. Huang, M. Ren, and J. Yu, “Software vulnerability analysis across programming language and program representation landscapes: A survey,” 2025

    2025

  44. [53]

    Systematic review: Analysis of coding vulnerabilities across languages,

    S. Sakharkar, “Systematic review: Analysis of coding vulnerabilities across languages,”Journal of Information Security, vol. 14, pp. 330– 342, 2023

    2023

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.