AutoResearchBench is a new benchmark showing top AI agents achieve under 10% success on complex scientific literature discovery tasks that demand deep comprehension and open-ended search.
hub
Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan
16 Pith papers cite this work. Polarity classification is still indexing.
16
Pith papers citing it
hub tools
citation-role summary
background 4
citation-polarity summary
roles
background 4representative citing papers
PixJail automates construction of paper-specific attack modules and unified evaluation pipelines for text-to-image jailbreaks, reproducing eleven methods with 2.1% average and 0% median error.
Paper-replication makes coding agents complete SciML paper replications only when every recorded claim has provenance, comparison evidence, and report coverage in a validated workspace.
Proposes agentic framework-based reproduction with a slot-binding interface to turn 16 PHM papers into standardized, assumption-aware benchmark implementations.
VeriCache turns lossy KV cache compression into lossless LLM inference by drafting with compressed cache and verifying drafts with full cache, achieving up to 4x throughput with identical outputs.
MLReplicate benchmark evaluates six autonomous systems on 45 manuscripts from ICML 2025 papers, finding that automated reviews accept flawed outputs with fabricated claims while human review exposes methodological failures, and that the cheapest system outperforms the most expensive by a wide margin
LLM-generated implementations of TNO spectral reconstruction from photometry exhibit an entropy floor of divergent code even after full methods text is provided, as LLMs recover core structure but miss tacit calibration knowledge.
ARA uses LLMs to build workflow graphs linking sources, methods, and outputs in papers, then scores reproducibility, reaching ~61% accuracy on 213 ReScience C articles and outperforming priors on ReproBench and GoldStandardDB.
HiRAS introduces hierarchical multi-agent coordination for paper-to-code generation and experiment reproduction, claiming over 10% relative gains over prior state-of-the-art on a refined benchmark with reduced hallucination.
PosterForest uses a hierarchical Poster Tree and content-layout agent collaboration to generate scientific posters from papers without training, outperforming prior automated methods in human and automated evaluation.
RExBench is a new benchmark showing that LLM coding agents fail to autonomously implement most realistic research extensions to prior AI papers.
Introduces the Agentic Publication Protocol (APP) as a repository-based standard for publishing papers together with reproducibility artifacts and agent instructions.
Proposes a code-first peer review protocol using AI infrastructure to execute research artifacts and generate claim-evidence review packages for human reviewers.
AblateCell reproduces baselines in three single-cell perturbation repositories with 88.9% success and recovers ground-truth critical components with 93.3% accuracy via closed-loop ablation.
AI can generate research artifacts faster than it can verify them, so across all eight lifecycle stages the credible deployment mode is human-governed collaboration rather than full autonomy.
citing papers explorer
-
AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
AutoResearchBench is a new benchmark showing top AI agents achieve under 10% success on complex scientific literature discovery tasks that demand deep comprehension and open-ended search.
-
PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation
PixJail automates construction of paper-specific attack modules and unified evaluation pipelines for text-to-image jailbreaks, reproducing eleven methods with 2.1% average and 0% median error.
-
Coding-agents can replicate scientific machine learning papers
Paper-replication makes coding agents complete SciML paper replications only when every recorded claim has provenance, comparison evidence, and report coverage in a validated workspace.
-
From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence
Proposes agentic framework-based reproduction with a slot-binding interface to turn 16 PHM papers into standardized, assumption-aware benchmark implementations.
-
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
VeriCache turns lossy KV cache compression into lossless LLM inference by drafting with compressed cache and verifying drafts with full cache, achieving up to 4x throughput with identical outputs.
-
MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility
MLReplicate benchmark evaluates six autonomous systems on 45 manuscripts from ICML 2025 papers, finding that automated reviews accept flawed outputs with fabricated claims while human review exposes methodological failures, and that the cheapest system outperforms the most expensive by a wide margin
-
Quantifying the Reconstructability of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction
LLM-generated implementations of TNO spectral reconstruction from photometry exhibit an entropy floor of divergent code even after full methods text is provided, as LLMs recover core structure but miss tacit calibration knowledge.
-
ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review
ARA uses LLMs to build workflow graphs linking sources, methods, and outputs in papers, then scores reproducibility, reaching ~61% accuracy on 213 ReScience C articles and outperforming priors on ReproBench and GoldStandardDB.
-
HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution
HiRAS introduces hierarchical multi-agent coordination for paper-to-code generation and experiment reproduction, claiming over 10% relative gains over prior state-of-the-art on a refined benchmark with reduced hallucination.
-
PosterForest: Hierarchical Multi-Agent Collaboration for Scientific Poster Generation
PosterForest uses a hierarchical Poster Tree and content-layout agent collaboration to generate scientific posters from papers without training, outperforming prior automated methods in human and automated evaluation.
-
RExBench: Can coding agents autonomously implement AI research extensions?
RExBench is a new benchmark showing that LLM coding agents fail to autonomously implement most realistic research extensions to prior AI papers.
-
Agentic Publication Protocol: An Attempt to Modernize Scientific Publication
Introduces the Agentic Publication Protocol (APP) as a repository-based standard for publishing papers together with reproducibility artifacts and agent instructions.
-
Review the Code, Not the Story: A Vision and Protocol for Code-First Peer Review
Proposes a code-first peer review protocol using AI infrastructure to execute research artifacts and generate claim-evidence review packages for human reviewers.
-
AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositories
AblateCell reproduces baselines in three single-cell perturbation repositories with 88.9% success and recovers ground-truth critical components with 93.3% accuracy via closed-loop ablation.
-
AI for Auto-Research: Roadmap & User Guide
AI can generate research artifacts faster than it can verify them, so across all eight lifecycle stages the credible deployment mode is human-governed collaboration rather than full autonomy.
- ArtifactLinker: Linking Scientific Artifacts for Automatic State-of-the-Art Discovery