REVIEW 6 cited by
MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In software development, resolving the emergent issues within GitHub repositories is a complex challenge that involves not only the incorporation of new code but also the maintenance of existing code. Large Language Models (LLMs) have shown promise in code generation but face difficulties in resolving Github issues, particularly at the repository level. To overcome this challenge, we empirically study the reason why LLMs fail to resolve GitHub issues and analyze the major factors. Motivated by the empirical findings, we propose a novel LLM-based Multi-Agent framework for GitHub Issue reSolution, MAGIS, consisting of four agents customized for software evolution: Manager, Repository Custodian, Developer, and Quality Assurance Engineer agents. This framework leverages the collaboration of various agents in the planning and coding process to unlock the potential of LLMs to resolve GitHub issues. In experiments, we employ the SWE-bench benchmark to compare MAGIS with popular LLMs, including GPT-3.5, GPT-4, and Claude-2. MAGIS can resolve 13.94% GitHub issues, significantly outperforming the baselines. Specifically, MAGIS achieves an eight-fold increase in resolved ratio over the direct application of GPT-4, the advanced LLM.
Forward citations
Cited by 6 Pith papers
-
From Static Analysis to Audience Dissemination: A Training-Free Multimodal Controversy Detection Multi-Agent Framework
AuDisAgent reformulates multimodal controversy detection as a dynamic audience dissemination process using screening, panel discussion, and arbitration agents, plus comment bootstrapping, and reports outperforming pri...
-
Evaluating LLM Agents on Automated Software Analysis Tasks
A custom LLM agent achieves 94% manually verified success on a new benchmark of 35 software analysis setups, outperforming baselines at 77%, but struggles with stage mixing, error localization, and overestimating its ...
-
Evaluating LLM Agents on Automated Software Analysis Tasks
A purpose-built, staged LLM agent correctly sets up and executes software analysis tools on 33 of 35 benchmark tasks, outperforming general-purpose agent baselines by at least 17 percentage points.
-
Emergent Coordination in Multi-Agent Language Models
Multi-agent LLM systems can be steered via prompt design from mere aggregates to higher-order collectives with identity-linked differentiation and goal-directed complementarity, as measured by partial information deco...
-
IFEvalCode: Controlled Code Generation
A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.
-
Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair
On 372 multi-hunk bugs, coding agents repair between 26% and 93% of defects, accuracy falls as edits become more divergent and dispersed, and failed repairs consume up to 343% more tokens.
Discussion (0). Sign in to comment.