REVIEW 13 cited by
R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Improving open-source models on real-world SWE tasks (solving GITHUB issues) faces two key challenges: 1) scalable curation of execution environments to train these models, and, 2) optimal scaling of test-time compute. We introduce AgentGym, the largest procedurally-curated executable gym environment for training real-world SWE-agents, consisting of more than 8.7K tasks. AgentGym is powered by two main contributions: 1) SYNGEN: a synthetic data curation recipe that enables scalable curation of executable environments using test-generation and back-translation directly from commits, thereby reducing reliance on human-written issues or unit tests. We show that this enables more scalable training leading to pass@1 performance of 34.4% on SWE-Bench Verified benchmark with our 32B model. 2) Hybrid Test-time Scaling: we provide an in-depth analysis of two test-time scaling axes; execution-based and execution-free verifiers, demonstrating that they exhibit complementary strengths and limitations. Test-based verifiers suffer from low distinguishability, while execution-free verifiers are biased and often rely on stylistic features. Surprisingly, we find that while each approach individually saturates around 42-43%, significantly higher gains can be obtained by leveraging their complementary strengths. Overall, our approach achieves 51% on the SWE-Bench Verified benchmark, reflecting a new state-of-the-art for open-weight SWE-agents and for the first time showing competitive performance with proprietary models such as o1, o1-preview and sonnet-3.5-v2 (with tools). We will open-source our environments, models, and agent trajectories.
Forward citations
Cited by 13 Pith papers
-
Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First
A 7B searcher's sandbox-verified handoff lets a cheap coding model match a frontier model's solve rate on SWE-bench Pro at roughly one fifth the cost, while the router that picks the fixer contributes nothing on this ...
-
Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks
A prefix-only 0.6B monitor can forecast SWE-agent failure and, paired with an edit-overlay restart, reclaim tokens and improve resolution.
-
RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement
A controlled benchmark shows LLM agents can sometimes discover better training-data strategies through feedback, but their improvements are fragile and usually not sustained.
-
NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs
Requirement-driven synthesis generates executable agent training tasks and trajectories from a capability description, improving a 35B model from 22.5% to 52.0% on Terminal-Bench 2.0 and from 813 to 1338 GDPval Elo.
-
SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review
An agentic code reviewer that explores repositories to judge and diagnose AI-generated pull requests improves resolve rates from 27.5% to 56.9% and outperforms single-turn review baselines.
-
BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
A 500-task benchmark shows AI code agents plateau around 56% accuracy on broader software tasks, with search access providing limited and inconsistent gains.
-
SERA: Soft-Verified Efficient Repository Agents
SERA trains competitive coding agents via supervised finetuning on synthetic trajectories verified only by patch-overlap recall, cutting data-generation cost by roughly two orders of magnitude.
-
Toward Training Superintelligent Software Agents through Self-Play SWE-RL
Self-play RL on bug injection and repair in sandboxed repositories yields +10.4 and +7.8 point gains on SWE-bench Verified and Pro while outperforming human-data baselines.
-
Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering
A 32B model trained to self-evolve patch candidates through RL reaches 41.6% on SWE-Bench Verified with 50 samples, comparable to a 70B model using 500 samples.
-
Process Reward Informed Tree Rollout for Effective Multi-Turn RL
Using process feedback to guide which partial trajectories to branch on, PATR replaces uniform independent rollouts with adaptive tree rollouts for multi-turn agent RL, reporting +5.0 points on SWE-Bench and +9.3 on F...
-
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Turn-level rewards from TD changes in a frozen reference model's gold-answer log-probability improve long-horizon search-agent RL on closed- and open-web benchmarks.
-
LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent
LiteResearcher uses a lite virtual world to make agentic RL training scalable and stable, enabling a 4B model to achieve 71.3% on GAIA and 78.0% on Xbench, outperforming larger open-source and commercial systems.
-
Self-Evolving Coding Agents
A systematic survey that defines self-evolving coding agents, organizes systems into a five-category object-centered taxonomy, and analyzes when and on what evidence these agents evolve.
Discussion (0). Continue with ORCID to comment.