REVIEW 18 cited by
Self-collaboration Code Generation via ChatGPT
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Although Large Language Models (LLMs) have demonstrated remarkable code-generation ability, they still struggle with complex tasks. In real-world software development, humans usually tackle complex tasks through collaborative teamwork, a strategy that significantly controls development complexity and enhances software quality. Inspired by this, we present a self-collaboration framework for code generation employing LLMs, exemplified by ChatGPT. Specifically, through role instructions, 1) Multiple LLM agents act as distinct `experts', each responsible for a specific subtask within a complex task; 2) Specify the way to collaborate and interact, so that different roles form a virtual team to facilitate each other's work, ultimately the virtual team addresses code generation tasks collaboratively without the need for human intervention. To effectively organize and manage this virtual team, we incorporate software-development methodology into the framework. Thus, we assemble an elementary team consisting of three LLM roles (i.e., analyst, coder, and tester) responsible for software development's analysis, coding, and testing stages. We conduct comprehensive experiments on various code-generation benchmarks. Experimental results indicate that self-collaboration code generation relatively improves 29.9%-47.1% Pass@1 compared to the base LLM agent. Moreover, we showcase that self-collaboration could potentially enable LLMs to efficiently handle complex repository-level tasks that are not readily solved by the single LLM agent.
Forward citations
Cited by 18 Pith papers
-
Rethinking Repetition Problems of LLMs in Code Generation
RPG penalizes tokens that continue repeated grammar structures in LLM-generated code, reducing structural repetition and improving Pass@k on HumanEval and MBPP.
-
Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
An interactive code generation pipeline that uses editable inline comments as a feedback channel improves pass@1 on HumanEval/MBPP and user task success in a small study.
-
AutoBridge: Automating Smart Device Integration with Centralized Platform
AutoBridge generates platform-compliant IoT integration code with 93.87% success and 94.87% function coverage in fully automatic mode, reaching 100% coverage with minimal yes/no human feedback.
-
Single-agent or Multi-agent Systems? Why Not Both?
On 15 agentic benchmarks, the accuracy advantage of multi-agent LLM systems over single-agent systems mostly disappears with stronger base models, and a hybrid single/multi-agent cascade improves accuracy and cuts cost.
-
A Solver-Aided Hierarchical Language for LLM-Driven CAD Design
A solver-aided hierarchical DSL lets an untuned LLM generate precise, editable 2D CAD geometry from text prompts, outperforming OpenSCAD slightly on CLIP alignment.
-
Do as We Do, Not as You Think: the Conformity of Large Language Models
In multi-agent question answering, all 11 tested LLMs abandon some correct answers to follow a unanimous wrong majority, with the effect strongest under the Doubt protocol.
-
Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash
A Balderdash simulation framework shows LLMs generate plausible fake definitions but fail to reason over game rules or adapt strategy, with the effect strongest on rare words.
-
Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report
A mixed-method experience report on LLM-based multi-agent frameworks for software engineering: broad feature coverage, weak monitoring support, and no clear quality winner on a README-summarization task, with incomple...
-
Facilitating Video Story Interaction with Multi-Agent Collaborative System
A multi-agent system with VLM and RAG lets users talk with stage-aware Harry Potter characters and customize scenes, with a user study reporting enhanced engagement.
-
Revisit Self-Debugging with Self-Generated Tests for Code Generation
Post-execution self-debugging with self-generated tests degrades performance on basic code tasks due to biased test labels, while in-execution trace-based self-debugging mitigates the bias and improves pass rates.
-
Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency
Chaining a simple two-agent coding workflow with execution-based debugging improves code-generation accuracy by about 0.7 percent over debugging alone, but the improvement is not statistically robust.
-
Code2API: A Tool for Generating Reusable APIs from Stack Overflow Code Snippets
Code2API prompts GPT-3.5-turbo with chain-of-thought and few-shot examples to turn Stack Overflow snippets into reusable APIs, and reports higher quality than the rule-based APIzator and near-human performance.
-
Are AI agents the new machine translation frontier? Challenges and opportunities of single- and multi-agent systems for multilingual digital communication
In a single-document pilot, a four-agent LLM translation workflow scored higher on adequacy and fluency than DeepL or Google Translate for English-Spanish legal text, but the result lacks statistical support and a sin...
-
CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation
CodeCoR improves LLM code generation by having four agents generate, test, prune, and repair candidate solutions, reporting a 77.8% average Pass@1 on four Python benchmarks.
-
The Current Challenges of Software Engineering in the Era of Large Language Models
The paper reports 26 challenges in LLM-based software engineering, grouped into seven aspects, derived from a structured discussion among 24 academics and practitioners.
-
From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents
A structured survey that categorizes LLM-based social simulation into individual, scenario, and society simulation, with associated methods, benchmarks, and observed trends.
-
WiS Platform: Enhancing Evaluation of LLM-Based Multi-Agent Systems Through Game-Based Analysis
An open platform and benchmark for evaluating LLM-based multi-agent systems through the 'Who is Spy?' game, including a leaderboard and behavioral analysis of ten models.
-
Exploration of LLM Multi-Agent Application Implementation Based on LangGraph+CrewAI
The paper describes, but does not measure, an integration of LangGraph and CrewAI for building multi-agent LLM applications.
Discussion (0). Continue with ORCID to comment.