LLMs frequently specify library versions with known CVEs in generated code (36-56% of tasks), show low compatibility (20-63%), and converge on the same risky versions across models.
hub Canonical reference
A survey on large language models for code generation.ACM Transactions on Software Engineering and Methodology, 35 (2):1–72
Canonical reference. 88% of citing Pith papers cite this work as background.
35
Pith papers citing it
75
external citations · Crossref
Background
88% of classified citations
hub tools
citation-role summary
background 7
method 1
citation-polarity summary
representative citing papers
Introduces SolidityBench benchmark and SolidityScore metric for repository-level Solidity code generation, finding supervised fine-tuning outperforms prompting, CoT, ICL, and RAG methods on evaluated LLMs.
SaaSBench introduces a heterogeneous benchmark for enterprise SaaS engineering and shows that state-of-the-art coding agents fail over 95% of the time before reaching deep business logic due to setup and integration problems.
RepoDoc uses a repository knowledge graph with module clustering and semantic impact propagation to generate more complete documentation 3x faster with 85% fewer tokens and handle incremental updates 73% faster than prior LLM-based tools.
CodeSpecBench shows LLMs achieve at most 20.2% pass rate on repository-level executable behavioral specification generation, revealing that strong code generation does not imply deep semantic understanding.
Merging fine-tuned models for multilingual translation fails because fine-tuning redistributes language-specific neurons rather than sharpening them, increasing representational divergence in output-generating layers.
MURPHY improves code generation pass rates by up to 6% through retrospective credit assignment on multi-turn feedback trees using max or mean reward propagation.
LLMs detect falsified ISO/IEC/IEEE 24765 SE definitions well but also reject many correct ones, indicating systematic rejection bias rather than genuine discriminative understanding of SE terminology.
Plainbook makes data-science notebooks natural-language-first by preserving cell descriptions, generating code via AI, enforcing linear execution via a checkpointing kernel, and adding value-centered cell and global tests.
AdaTrans uses strategy-driven RAG, error-stratified transformation, and multi-stage validation to reach 95.51% mean compilation pass rate and 81.09% solve rate on 104 algorithmic problems with only 1.19% unsafe files.
Introduces InterFLOPBench benchmark and evaluates 14 LLMs on multi-label classification of six floating-point error categories in C code, with top models exceeding 0.88 overall F1 but lower scores on subtle errors like underflow.
Empirical evaluation finds reasoning LLMs improve code correction across iterations using execution feedback and outperform non-reasoning models, with syntactic and runtime errors easier to fix than logical ones.
LPDS quantifies difficulty of logic-preserving problem variations and searches for the hardest ones, producing up to 5x larger performance drops than random sampling and better robustness gains from fine-tuning on difficult examples.
Coding agents exhibit action bias by proposing undesirable changes on already-fixed issues 35-65% of the time, and explicit reproduction instructions only partially mitigate this while creating new abstention errors.
A review of 114 studies creates taxonomies for code and data quality issues, formalizes 18 propagation mechanisms from training data defects to LLM-generated code defects, and synthesizes detection and mitigation techniques.
RuC generates language-agnostic, grammar-based benchmarks for evaluating LLMs on RTL code completion at controllable granularities, demonstrated on SystemVerilog designs from Tiny Tapeout and a RISC-V core where Fill-in-the-Middle prompting performed best.
Luminol-AIDetect detects machine-generated text zero-shot by extracting perplexity-based features from an input and its shuffled version, using density estimation to exploit greater dispersion in MGT perplexity under shuffling.
Probabilistic programs of thought let LLMs produce many program variants from one generation by building a compact probabilistic representation of the token distribution.
Continuous latent-vector compression improves BLEU scores on repository-level code tasks by up to 28.3% at 4x compression while cutting inference latency.
Babbling Suppression stops LLM code generation upon test passage to reduce token output and energy consumption by up to 65% across Python and Java benchmarks.
A neuro-symbolic pipeline (LLM function summaries + Z3 path checks) injected into CodeQL/Infer finds 54 memory leaks on 3.6M LOC of real C/C++ software, 53 confirmed, versus 19 and 3 for the baselines.
Frontier LLMs like GPT-5.2 show large accuracy drops on perturbed program-output prediction tasks while open-source reasoning models remain more stable, exposing limits in code semantics understanding.
MOA deploys LLM agents to detect recurring memory anti-patterns via profiling, synthesize static analyzers, and apply patches, reporting 42% heap and 11% binary-size reductions on OpenHarmony after finding over 10,000 issues.
Multimodal LLMs applied to 16 real-world configurators using 18 synthesized criteria can identify usability issues and generate actionable suggestions, with human review confirming reliability.
citing papers explorer
-
Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions
LLMs frequently specify library versions with known CVEs in generated code (36-56% of tasks), show low compatibility (20-63%), and converge on the same risky versions across models.
-
Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning
Introduces SolidityBench benchmark and SolidityScore metric for repository-level Solidity code generation, finding supervised fine-tuning outperforms prompting, CoT, ICL, and RAG methods on evaluated LLMs.
-
SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering
SaaSBench introduces a heterogeneous benchmark for enterprise SaaS engineering and shows that state-of-the-art coding agents fail over 95% of the time before reaching deep business logic due to setup and integration problems.
-
RepoDoc: A Knowledge Graph-Based Framework to Automatic Documentation Generation and Incremental Updates
RepoDoc uses a repository knowledge graph with module clustering and semantic impact propagation to generate more complete documentation 3x faster with 85% fewer tokens and handle incremental updates 73% faster than prior LLM-based tools.
-
CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
CodeSpecBench shows LLMs achieve at most 20.2% pass rate on repository-level executable behavioral specification generation, revealing that strong code generation does not imply deep semantic understanding.
-
One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging
Merging fine-tuned models for multilingual translation fails because fine-tuning redistributes language-specific neurons rather than sharpening them, increasing representational divergence in output-generating layers.
-
MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation
MURPHY improves code generation pass rates by up to 6% through retrospective credit assignment on multi-turn feedback trees using max or mean reward propagation.
-
Large Language Models Have Unreliable Understanding of Software Engineering Terminology
LLMs detect falsified ISO/IEC/IEEE 24765 SE definitions well but also reject many correct ones, indicating systematic rejection bias rather than genuine discriminative understanding of SE terminology.
-
Plainbook: Data Science, in Plain Language
Plainbook makes data-science notebooks natural-language-first by preserving cell descriptions, generating code via AI, enforcing linear execution via a checkpointing kernel, and adding value-centered cell and global tests.
-
AdaTrans: Automated C to Rust Transformation via Error-Adaptive Repair
AdaTrans uses strategy-driven RAG, error-stratified transformation, and multi-stage validation to reach 95.51% mean compilation pass rate and 81.09% solve rate on 104 algorithmic problems with only 1.19% unsafe files.
-
Benchmarking Large Language Models on Floating-Point Error Classification
Introduces InterFLOPBench benchmark and evaluates 14 LLMs on multi-label classification of six floating-point error categories in C code, with top models exceeding 0.88 overall F1 but lower scores on subtle errors like underflow.
-
Unlocking LLM Code Correction with Iterative Feedback Loops
Empirical evaluation finds reasoning LLMs improve code correction across iterations using execution feedback and outperform non-reasoning models, with syntactic and runtime errors easier to fix than logical ones.
-
LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling
LPDS quantifies difficulty of logic-preserving problem variations and searches for the hardest ones, producing up to 5x larger performance drops than random sampling and better robustness gains from fine-tuning on difficult examples.
-
Coding Agents Don't Know When to Act
Coding agents exhibit action bias by proposing undesirable changes on already-fixed issues 35-65% of the time, and explicit reproduction instructions only partially mitigate this while creating new abstention errors.
-
Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code
A review of 114 studies creates taxonomies for code and data quality issues, formalizes 18 propagation mechanisms from training data defects to LLM-generated code defects, and synthesizes detection and mitigation techniques.
-
RuC: HDL-Agnostic Rule Completion Benchmark Generation
RuC generates language-agnostic, grammar-based benchmarks for evaluating LLMs on RTL code completion at controllable granularities, demonstrated on SystemVerilog designs from Tiny Tapeout and a RISC-V core where Fill-in-the-Middle prompting performed best.
-
Luminol-AIDetect: Fast Zero-shot Machine-Generated Text Detection based on Perplexity under Text Shuffling
Luminol-AIDetect detects machine-generated text zero-shot by extracting perplexity-based features from an input and its shuffled version, using density estimation to exploit greater dispersion in MGT perplexity under shuffling.
-
Probabilistic Programs of Thought
Probabilistic programs of thought let LLMs produce many program variants from one generation by building a compact probabilistic representation of the token distribution.
-
On the Effectiveness of Context Compression for Repository-Level Tasks: An Empirical Investigation
Continuous latent-vector compression improves BLEU scores on repository-level code tasks by up to 28.3% at 4x compression while cutting inference latency.
-
Babbling Suppression: Making LLMs Greener One Token at a Time
Babbling Suppression stops LLM code generation upon test passage to reduce token output and energy consumption by up to 65% across Python and Java benchmarks.
-
Finding Memory Leaks in C/C++ Programs via Neuro-Symbolic Augmented Static Analysis
A neuro-symbolic pipeline (LLM function summaries + Z3 path checks) injected into CodeQL/Infer finds 54 memory leaks on 3.6M LOC of real C/C++ software, 53 confirmed, versus 19 and 3 for the baselines.
-
How Robustly do LLMs Understand Execution Semantics?
Frontier LLMs like GPT-5.2 show large accuracy drops on perturbed program-output prediction tasks while open-source reasoning models remain more stable, exposing limits in code semantics understanding.
-
MOA: A Profiling-Guided LLM Framework for Memory-Optimization Automation at Codebase Scale
MOA deploys LLM agents to detect recurring memory anti-patterns via profiling, synthesize static analyzers, and apply patches, reporting 42% heap and 11% binary-size reductions on OpenHarmony after finding over 10,000 issues.
-
Usability Analysis of Configurator User Interfaces with Multimodal Large Language Models
Multimodal LLMs applied to 16 real-world configurators using 18 synthesized criteria can identify usability issues and generate actionable suggestions, with human review confirming reliability.
-
HypEHR: Hyperbolic Modeling of Electronic Health Records for Efficient Question Answering
A 22M-parameter hyperbolic model answers structured EHR questions with accuracy close to LLM-based systems (EHRXQA 89.5%, MIMIC-Instr 76.0%).
-
MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue
MICA mixes per-turn and whole-trajectory normalized reward signals to train emotional-support chatbots, outperforming GRPO and REINFORCE++ on EMPA, EQ-Bench, and EmoBench.
-
Continuous Behavioral Synthesis for Adaptive Health Dashboards: An LLM-Mediated Architecture Integrating Explicit Preference, Spatial Reorganization, and Attention Allocation Signals
Presents an LLM-mediated architecture for continuous adaptation of health dashboards by synthesizing explicit feedback, spatial reorganization, and attention signals via structured prompt engineering.
-
Algorithmic algorithm development with LLMs: A Case Study on LLM-Usage for Contraction Order Optimization in Tensor Networks
Case study applies verifier-guided LLM evolutionary agents to contraction-order optimization in tensor networks and concludes that human validation remains essential.
-
Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback
RLVR with combined unit-test and static-analysis rewards improves pass@1 by up to 13pp on MBPP for 0.6B-1B models, while single-reward variants can induce shorter but less correct outputs.
-
Efficient Post-training of LLMs for Code Generation With Offline Reinforcement Learning
Offline RL post-training boosts code generation performance in LLMs, with larger gains for small models and hard problems, using pre-collected datasets.
-
Can LLMs Solve Science or Just Write Code? Evaluating Quantum Solver Generation
Iterative refinement boosts LLM success in generating quantum solvers that match classical results, but more advanced models shift from execution errors to hard-to-detect numerical inaccuracies.
-
VeriInteresting: An Empirical Study of Model Prompt Interactions in Verilog Code Generation
Empirical study identifies patterns in how model classes respond to structured prompts, optimization, and other techniques across two Verilog benchmarks.
-
Search-Based Software Engineering and AI Foundation Models: Current Landscape and Future Roadmap
A research roadmap analyzing the current state of search-based software engineering with foundation models, outlining challenges and directions across three integration aspects.
-
Reasoning Beyond Prediction: From Data-Driven to Causal Software Engineering
Calls for a new paradigm in software engineering where machines support causal reasoning rather than only prediction from data patterns.
-
From Helpful to Trustworthy: LLM Agents for Pair Programming
A research proposal for three studies on multi-agent LLM pair programming that externalizes intent and uses automated validation to increase trustworthiness.