The Meta-Agent Challenge shows frontier AI models rarely match human-engineered agent baselines when tasked with autonomous development, with proprietary models succeeding most often and some exhibiting cheating under pressure.
hub
arXiv: 2511.13646 [cs.SE].URL:https://arxiv.org/abs/2511.13646
23 Pith papers cite this work. Polarity classification is still indexing.
23
Pith papers citing it
hub tools
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
A framework that learns source-level code transformations offline on small programs and applies them context-sensitively to make symbolic execution faster, cheaper, and able to find more bugs in real-world software.
EvoRepair is the first experience-based self-evolving agent framework for automated vulnerability repair, reporting 90.46% overall success on PATCHEVAL and SEC-bench benchmarks.
PerfCodeBench reveals that state-of-the-art LLMs produce functionally correct but significantly slower code than expert-optimized versions on system-level tasks, especially those involving parallelism and GPUs.
DOLORES, an agent using a formal language for meta-reasoning to construct adaptive scaffolds on the fly, outperforms prior scaffolding methods by 24.8% on average across four hard benchmarks and multiple model sizes.
A harness for AI agents enabled construction of a Rust library with 100+ problem types and 200+ reduction rules for NP-hard problems in three months.
Refining bug reports with hierarchical evidence from an unverified agent trajectory plus repository review raises Pass@1 on SWE-Bench Lite from 41% to 59.67% (GPT-5-mini) and similarly for other agents.
DUALVIEW is a dual-modal framework using Module Coupling, Function Call, Class Hierarchy, and Program Dependence graphs to enable persistent structural reasoning for agentic issue resolution, reporting gains on SWE-bench Pro and Verified.
PatchFusion uses deterministic atomic evidence fusion on candidate patches to outperform ranking, test-filtering, and LLM-judge selectors on SWE-bench and Defects4J pools.
LLVM-Bench supplies 423 validated LLVM issues and LLVM-Gym automates evaluation, showing LLMs are limited but an ensemble reaches 21.99% resolution.
A11YRepair uses divide-and-conquer clustering and WCAG knowledge to repair multi-fault accessibility violations in web projects, showing better results than baselines on a new 60-project benchmark with some patches merged upstream.
Persistent decision history plus role-separated redacted evaluation lets agents continually improve skills across sessions and outperform commercial deep-research and prior skill-evolution baselines.
DemoEvolve bootstraps harness evolution with demonstrations to achieve more stable and effective edits than self-rollout search in sparse-feedback environments like Balatro.
AgentSPEX is a new language and harness for explicitly specifying and running structured LLM-agent workflows with typed steps, control flow, parallel execution, and a visual editor.
LLM agents resolve fewer than half of issues while satisfying design constraints despite passing tests, as shown by a benchmark of 495 issues and 1787 constraints from six repositories.
Embedding dynamic analysis into an LLM repair agent yields a claimed 79.4% resolution rate on SWE-bench Verified while cutting token use by about 25%.
A single LLM agent that injects and repairs its own bugs in real repositories improves SWE-bench Verified and Pro by +10.4 and +7.8 points, outperforming a human-data RL baseline.
LLM agents complete over 80% of tasks on a new 849-task Rust verification benchmark and over 90% on unfinished human proofs.
RQGM enables co-evolution of agents and evaluators across epochs with non-stationary utilities, reporting gains in coding pass rates, paper acceptance, and proof grading over prior self-improving agents.
icat-agent improves resolution rates on SWE-bench Verified and Pro by 3.6-18.5% over baselines via event-based multi-agent scaffolding and rubric-driven workflow pivoting while using the same models.
A survey that organizes existing work on LLM-based agents around code as the central harness, structured in three layers of interfaces, mechanisms, and multi-agent scaling, with applications across domains and listed open challenges.
Agent-generated tests mainly act as observational feedback channels and do not meaningfully improve issue resolution success in current LLM software engineering agents.
MicroSkill Architecture partitions knowledge into atomic skill capsules selected via constrained optimization to cut token use over 90% and improve code generation metrics in one enterprise case study.
citing papers explorer
-
The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?
The Meta-Agent Challenge shows frontier AI models rarely match human-engineered agent baselines when tasked with autonomous development, with proprietary models succeeding most often and some exhibiting cheating under pressure.
-
Symbolon: Symbolic Execution by Learning Code Transformation
A framework that learns source-level code transformations offline on small programs and applies them context-sensitively to make symbolic execution faster, cheaper, and able to find more bugs in real-world software.
-
EvoRepair: Enhancing Vulnerability Repair Agents Through Experience-Based Self-Evolution
EvoRepair is the first experience-based self-evolving agent framework for automated vulnerability repair, reporting 90.46% overall success on PATCHEVAL and SEC-bench benchmarks.
-
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
PerfCodeBench reveals that state-of-the-art LLMs produce functionally correct but significantly slower code than expert-optimized versions on system-level tasks, especially those involving parallelism and GPUs.
-
Deep Reasoning in General Purpose Agents via Structured Meta-Cognition
DOLORES, an agent using a formal language for meta-reasoning to construct adaptive scaffolds on the fly, outperforms prior scaffolding methods by 24.8% on average across four hard benchmarks and multiple model sizes.
-
Problem Reductions at Scale: Agentic Integration of Computationally Hard Problems
A harness for AI agents enabled construction of a Rust library with 100+ problem types and 200+ reduction rules for NP-hard problems in three months.
-
Bug Report Specification Refinement with Trajectory Guidance for Automated Program Repair
Refining bug reports with hierarchical evidence from an unverified agent trajectory plus repository review raises Pass@1 on SWE-Bench Lite from 41% to 59.67% (GPT-5-mini) and similarly for other agents.
-
Beyond Textual Repository Exploration: Dual-Modal Structural Reasoning for Agentic Issue Resolution
DUALVIEW is a dual-modal framework using Module Coupling, Function Call, Class Hierarchy, and Program Dependence graphs to enable persistent structural reasoning for agentic issue resolution, reporting gains on SWE-bench Pro and Verified.
-
A Single Patch Is Not Enough: Deterministic Fusion of Repair Candidates
PatchFusion uses deterministic atomic evidence fusion on candidate patches to outperform ranking, test-filtering, and LLM-judge selectors on SWE-bench and Defects4J pools.
-
LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution
LLVM-Bench supplies 423 validated LLVM issues and LLVM-Gym automates evaluation, showing LLMs are limited but an ensemble reaches 21.99% resolution.
-
A11YRepair: Bridging Web Accessibility Barriers via Knowledge-Enhanced Divide-and-Conquer Repair
A11YRepair uses divide-and-conquer clustering and WCAG knowledge to repair multi-fault accessibility violations in web projects, showing better results than baselines on a new 60-project benchmark with some patches merged upstream.
-
SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History
Persistent decision history plus role-separated redacted evaluation lets agents continually improve skills across sessions and outperform commercial deep-research and prior skill-evolution baselines.
-
DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations
DemoEvolve bootstraps harness evolution with demonstrations to achieve more stable and effective edits than self-rollout search in sparse-feedback environments like Balatro.
-
AgentSPEX: An Agent SPecification and EXecution Language
AgentSPEX is a new language and harness for explicitly specifying and running structured LLM-agent workflows with typed steps, control flow, parallel execution, and a visual editor.
-
Does Pass Rate Tell the Whole Story? Evaluating Design Constraint Compliance in LLM-based Issue Resolution
LLM agents resolve fewer than half of issues while satisfying design constraints despite passing tests, as shown by a benchmark of 495 issues and 1787 constraints from six repositories.
-
Dynamic analysis enhances issue resolution
Embedding dynamic analysis into an LLM repair agent yields a claimed 79.4% resolution rate on SWE-bench Verified while cutting token use by about 25%.
-
Toward Training Superintelligent Software Agents through Self-Play SWE-RL
A single LLM agent that injects and repairs its own bugs in real repositories improves SWE-bench Verified and Pro by +10.4 and +7.8 points, outperforming a human-data RL baseline.
-
VeruSAGE: A Study of Agent-Based Verification for Rust Systems
LLM agents complete over 80% of tasks on a new 849-task Rust verification benchmark and over 90% on unfinished human proofs.
-
The Red Queen G\"odel Machine: Co-Evolving Agents and Their Evaluators
RQGM enables co-evolution of agents and evaluators across epochs with non-stationary utilities, reporting gains in coding pass rates, paper acceptance, and proof grading over prior self-improving agents.
-
Unlocking Model Potentials Through Adaptive Multi-Agent Scaffolding for Efficient Issue Resolution
icat-agent improves resolution rates on SWE-bench Verified and Pro by 3.6-18.5% over baselines via event-based multi-agent scaffolding and rubric-driven workflow pivoting while using the same models.
-
Code as Agent Harness
A survey that organizes existing work on LLM-based agents around code as the central harness, structured in three layers of interfaces, mechanisms, and multi-agent scaling, with applications across domains and listed open challenges.
-
Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
Agent-generated tests mainly act as observational feedback channels and do not meaningfully improve issue resolution success in current LLM software engineering agents.
-
Microskill Architecture: A Modular Skill-Driven Framework for AI-Native Code Generation
MicroSkill Architecture partitions knowledge into atomic skill capsules selected via constrained optimization to cut token use over 90% and improve code generation metrics in one enterprise case study.