REVIEW 20 cited by
From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the rise of large language models (LLMs), researchers are increasingly exploring their applications in var ious vertical domains, such as software engineering. LLMs have achieved remarkable success in areas including code generation and vulnerability detection. However, they also exhibit numerous limitations and shortcomings. LLM-based agents, a novel tech nology with the potential for Artificial General Intelligence (AGI), combine LLMs as the core for decision-making and action-taking, addressing some of the inherent limitations of LLMs such as lack of autonomy and self-improvement. Despite numerous studies and surveys exploring the possibility of using LLMs in software engineering, it lacks a clear distinction between LLMs and LLM based agents. It is still in its early stage for a unified standard and benchmarking to qualify an LLM solution as an LLM-based agent in its domain. In this survey, we broadly investigate the current practice and solutions for LLMs and LLM-based agents for software engineering. In particular we summarise six key topics: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance. We review and differentiate the work of LLMs and LLM-based agents from these six topics, examining their differences and similarities in tasks, benchmarks, and evaluation metrics. Finally, we discuss the models and benchmarks used, providing a comprehensive analysis of their applications and effectiveness in software engineering. We anticipate this work will shed some lights on pushing the boundaries of LLM-based agents in software engineering for future research.
Forward citations
Cited by 20 Pith papers
-
How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study
SE-agent development follows a recurring seven-stage loop where evaluation drives iteration, and challenges such as unreliable evaluation signals and comprehension debt emerge.
-
CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation
CrossPL, a 1,982-task benchmark built from GitHub repositories, shows that LLMs achieve at most 79.74% pass@1 on cross-language IPC code generation and struggle with low-level protocols like Pipe.
-
Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports
Introduces Active-SWE, a benchmark for proactive multi-bug fixing without issue reports, and shows state-of-the-art coding agents resolve few such bugs.
-
NeSyFS: A Neuro-symbolic Fast-Slow Thinking Framework for LLM Agent under Partial Observability
NeSyFS combines a knowledge-graph belief state, a reflection step, and TSMC-style particle planning to improve LLM agent success rates on ALFWorld, WebShop, and ScienceWorld.
-
Industrial Practice of LLM-Based Test Case Carving and Assertion Generation (Experience Paper)
NL2Test generates executable API regression tests from a natural-language scenario and a recorded traffic capture, achieving 82.4% exact-match on 51 industrial scenarios and 85.4% adoption across 3,196 deployed tests.
-
Bridging Behavior and Implementation: Automated Java Glue Code Generation for Behavior-Driven Development
AutoGlue, a behavior-first multi-agent LLM framework, generates Java BDD glue code that matches project APIs, with 46.1% of 1,307 outputs rated directly usable.
-
Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Code
Using a 61-feature readability model, LLM code matches or slightly exceeds human code in readability score, shows distinct issue patterns, and prompt design has limited influence.
-
Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms
Murakkab uses declarative workflow specs and a profile-guided MILP optimizer to reduce GPU, energy, and cost for agentic workflow serving while meeting percentile-defined SLOs.
-
Robot builds a robot's brain: AI generated drone command and control station hosted in the sky
An AI model generated a fully functional drone command-and-control website, deployed on the drone itself, and a real flight demonstrated it.
-
LLM-Driven Collaborative Model for Untangling Commits via Explicit and Implicit Dependency Reasoning
ColaUntangle, a three-agent LLM consultation framework with PDG-derived explicit and implicit contexts, reports 44% (C#) and 82% (Java) Accuracy_c gains over prior commit untangling baselines.
-
AutoData: A Multi-Agent System for Open Web Data Collection
AutoData, a multi-agent system with a hypergraph message cache, automates web dataset collection from a sentence instruction and outperforms general agent baselines on the new Instruct2DS benchmark.
-
HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization
HiLSVA shows that a human-in-the-loop LLM agent system can help novices and experts complete scientific visualization tasks, while human oversight adds measurable execution time.
-
Evaluating the Effectiveness of Large Language Models in Solving Simple Programming Tasks: A User-Centered Study
A within-subjects experiment with 15 high school students reports faster task completion with a collaborative ChatGPT-4o style than with a passive style, but the result is not significant versus proactive style and is...
-
Augmenting the Generality and Performance of Large Language Models for Software Engineering
A proposal for augmenting LLM generality in non-code software engineering tasks, with no reported experimental evidence.
-
CETBench: A Novel Dataset constructed via Transformations over Programs for Benchmarking LLMs for Code-Equivalence Checking
CETBench reveals that LLMs' code-equivalence accuracy drops sharply on program pairs modified by simple transformations, and fine-tuning on such pairs partly restores accuracy.
-
GPL-SLAM: A Laser SLAM Framework with Gaussian Process Based Extended Landmarks
A laser SLAM framework that models each object as a Gaussian-process contour, updated recursively and inferred jointly with the robot pose in a Bayesian framework.
-
Cognitive Agents Powered by Large Language Models for Agile Software Project Management
LLM agents acting as Agile roles produced plausible project artifacts in simulation, but the claimed improvements over human teams are unsupported because no comparison or validated metrics are provided.
-
On the Surprising Efficacy of LLMs for Penetration-Testing
A critical review arguing that LLMs are surprisingly effective for penetration testing because the task is largely pattern-matching, while noting serious reliability, safety, and cost barriers to autonomous use.
-
Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.
-
A Survey: Towards Privacy and Security in Mobile Large Language Models
A survey of privacy and security challenges for mobile large language models, summarizing known attack types and defenses without introducing new results.
Discussion (0). Sign in to comment.