Frontier coding agents surpass published Nature-family SOTA on only 17.8% of 90 sealed scientific tasks, mostly by recasting problems as supervised ML rather than inventing methods.
hub Canonical reference
Mlgym: A new framework and benchmark for advancing ai research agents
Canonical reference. 83% of citing Pith papers cite this work as background.
18
Pith papers citing it
4
external citations · external index
Background
83% of classified citations
hub tools
citation-role summary
background 4
dataset 2
citation-polarity summary
representative citing papers
A multi-agent prompt optimization system using environment feedback improves LLM agent success rates on BabyAI tasks from 0% to 72.5% on challenging coordination tasks.
TerraBench is a new benchmark with 403 tasks across Earth-science domains that evaluates LLM agents on coordinating heterogeneous data using executable ReAct-style workflows and process-level metrics.
SciAgentArena is a new interactive benchmark for AI agents on scientific tasks that finds agents handle clear data-analysis workflows but struggle with novel insights, self-directed exploration, and open-ended questions.
InquiTree shows LLM agents suffer from degrading critical capabilities during extended scientific interactions and perform worse on papers published after their training cutoffs.
Current AI agents fail to reliably invent transferable ML methods on a controlled 140-task benchmark, and more test-time compute or context does not remove the scientific-judgment bottleneck.
EO-Gym supplies an executable multimodal environment and 9k-trajectory benchmark that turns Earth Observation into a tool-using, multi-step reasoning task, revealing that current VLMs struggle on temporal and cross-sensor workflows while fine-tuning lifts Pass@3 from 0.49 to 0.74.
SAGE compares social co-evolution against matched self-evolution across three arenas and finds peer history enables breakthroughs only for agents that plateau under self-improvement, with abstraction of traces mattering more than raw volume.
MLReplicate benchmark evaluates six autonomous systems on 45 manuscripts from ICML 2025 papers, finding that automated reviews accept flawed outputs with fabricated claims while human review exposes methodological failures, and that the cheapest system outperforms the most expensive by a wide margin
Behavior Cue Reasoning trains LLMs to emit special tokens before behaviors, enabling monitors to cut up to 50% wasted reasoning tokens and recover safe actions from 80% of unsafe traces, more than doubling success rates with no performance cost.
TREX automates the LLM training lifecycle via collaborative agents and tree-based exploration, delivering consistent performance gains across 10 real-world fine-tuning tasks in FT-Bench.
Co-evolving LLM-generated solutions with their evaluators enables discovery of novel database algorithms that outperform state-of-the-art baselines, including a query rewrite policy with up to 6.8x lower latency.
A 40-task, 10-domain benchmark finds current auto-research agents and LLMs score only ~20–26 on re-discovering real published scientific artifacts.
HACO adapts MaskGIT from vision into MaskGXT with symmetry tokens and stratified sampling, reaching 79.06% METRe accuracy on MP-20 polymorph split versus 70.87% for the best baseline.
AgentCE-Bench is a lightweight grid-planning benchmark that controls task horizon via hidden slots H and difficulty via decoy budget B, validated across 13 models for consistent and discriminative evaluation.
A comprehensive review of self-evolving AI agents that improve themselves over time, organized via a framework of inputs, agent system, environment, and optimizers, with domain-specific and safety discussions.
AI can generate research artifacts faster than it can verify them, so across all eight lifecycle stages the credible deployment mode is human-governed collaboration rather than full autonomy.
A survey consolidating benchmarks, agent frameworks, real-world applications, and protocols for LLM-based autonomous agents into a proposed taxonomy with recommendations for future research.
citing papers explorer
-
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
Frontier coding agents surpass published Nature-family SOTA on only 17.8% of 90 sealed scientific tasks, mostly by recasting problems as supervised ML rather than inventing methods.
-
Environment-Grounded Automated Prompt Optimization for LLM Game Agents
A multi-agent prompt optimization system using environment feedback improves LLM agent success rates on BabyAI tasks from 0% to 72.5% on challenging coordination tasks.
-
TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data?
TerraBench is a new benchmark with 403 tasks across Earth-science domains that evaluates LLM agents on coordinating heterogeneous data using executable ReAct-style workflows and process-level metrics.
-
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
SciAgentArena is a new interactive benchmark for AI agents on scientific tasks that finds agents handle clear data-analysis workflows but struggle with novel insights, self-directed exploration, and open-ended questions.
-
InquiTree: Evaluating AI Agents in the Scientific Inquiry Loop with Paper-Derived Research Trees
InquiTree shows LLM agents suffer from degrading critical capabilities during extended scientific interactions and perform worse on papers published after their training cutoffs.
-
MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
Current AI agents fail to reliably invent transferable ML methods on a controlled 140-task benchmark, and more test-time compute or context does not remove the scientific-judgment bottleneck.
-
EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents
EO-Gym supplies an executable multimodal environment and 9k-trajectory benchmark that turns Earth Observation into a tool-using, multi-step reasoning task, revealing that current VLMs struggle on temporal and cross-sensor workflows while fine-tuning lifts Pass@3 from 0.49 to 0.74.
-
SAGE: A Quantitative Evaluation of Socialized Evolution in Agent Ecosystems
SAGE compares social co-evolution against matched self-evolution across three arenas and finds peer history enables breakthroughs only for agents that plateau under self-improvement, with abstraction of traces mattering more than raw volume.
-
MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility
MLReplicate benchmark evaluates six autonomous systems on 45 manuscripts from ICML 2025 papers, finding that automated reviews accept flawed outputs with fabricated claims while human review exposes methodological failures, and that the cheapest system outperforms the most expensive by a wide margin
-
Behavior Cue Reasoning: Monitorable Reasoning Improves Efficiency and Safety through Oversight
Behavior Cue Reasoning trains LLMs to emit special tokens before behaviors, enabling monitors to cut up to 50% wasted reasoning tokens and recover safe actions from 80% of unsafe traces, more than doubling success rates with no performance cost.
-
TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration
TREX automates the LLM training lifecycle via collaborative agents and tree-based exploration, delivering consistent performance gains across 10 real-world fine-tuning tasks in FT-Bench.
-
AI-Driven Research for Databases
Co-evolving LLM-generated solutions with their evaluators enables discovery of novel database algorithms that outperform state-of-the-art baselines, including a query rewrite policy with up to 6.8x lower latency.
-
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
A 40-task, 10-domain benchmark finds current auto-research agents and LLMs score only ~20–26 on re-discovering real published scientific artifacts.
-
Discovering Crystal Structure Prediction Algorithms with an AI Co-Scientist
HACO adapts MaskGIT from vision into MaskGXT with symmetry tokens and stratified sampling, reaching 79.06% METRe accuracy on MP-20 polymorph split versus 70.87% for the best baseline.
-
AgentCE-Bench: Agent Configurable Evaluation with Scalable Horizons and Controllable Difficulty under Lightweight Environments
AgentCE-Bench is a lightweight grid-planning benchmark that controls task horizon via hidden slots H and difficulty via decoy budget B, validated across 13 models for consistent and discriminative evaluation.
-
A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems
A comprehensive review of self-evolving AI agents that improve themselves over time, organized via a framework of inputs, agent system, environment, and optimizers, with domain-specific and safety discussions.
-
AI for Auto-Research: Roadmap & User Guide
AI can generate research artifacts faster than it can verify them, so across all eight lifecycle stages the credible deployment mode is human-governed collaboration rather than full autonomy.
-
From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review
A survey consolidating benchmarks, agent frameworks, real-world applications, and protocols for LLM-based autonomous agents into a proposed taxonomy with recommendations for future research.