Frontier coding agents surpass published Nature-family SOTA on only 17.8% of 90 sealed scientific tasks, mostly by recasting problems as supervised ML rather than inventing methods.
Fire-bench: Evaluating agents on the rediscovery of scientific insights
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
citation-role summary
background 3
citation-polarity summary
years
2026 7roles
background 3polarities
background 3representative citing papers
ReproRepo uses GitHub issues as natural supervision to benchmark LLM agents on detecting reproducibility blockers across 1,149 ML papers, with the top agent finding related issues for roughly 90% of cases.
Current AI agents fail to reliably invent transferable ML methods on a controlled 140-task benchmark, and more test-time compute or context does not remove the scientific-judgment bottleneck.
Closed-loop LM-agent auto research finds some transferable gains on molecular property prediction benchmarks via external data but shows non-transfer for model and feature edits selected on validation.
ScientistOne introduces Chain-of-Evidence and an audit system that achieves zero hallucinated references, perfect score verification, and top method-code alignment while matching or beating human experts on five frontier tasks and generalizing to six more.
CellScientist introduces a dual-space hierarchical orchestration system that enables closed-loop refinement of virtual cell models by routing execution discrepancies back to hypothesis or implementation updates, yielding improved benchmark performance with auditable traces.
A survey organizing AI-powered research automation into five workflow stages, defining AutoResearch and Vibe Research, and proposing five evaluation dimensions while noting domain-conditioned limits on autonomy.
citing papers explorer
-
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
Frontier coding agents surpass published Nature-family SOTA on only 17.8% of 90 sealed scientific tasks, mostly by recasting problems as supervised ML rather than inventing methods.
-
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
ReproRepo uses GitHub issues as natural supervision to benchmark LLM agents on detecting reproducibility blockers across 1,149 ML papers, with the top agent finding related issues for roughly 90% of cases.
-
MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
Current AI agents fail to reliably invent transferable ML methods on a controlled 140-task benchmark, and more test-time compute or context does not remove the scientific-judgment bottleneck.
-
Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements
Closed-loop LM-agent auto research finds some transferable gains on molecular property prediction benchmarks via external data but shows non-transfer for model and feature edits selected on validation.
-
ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence
ScientistOne introduces Chain-of-Evidence and an audit system that achieves zero hallucinated references, perfect score verification, and top method-code alignment while matching or beating human experts on five frontier tasks and generalizing to six more.
-
CellScientist: Dual-Space Hierarchical Orchestration for Closed-Loop Refinement of Virtual Cell Models
CellScientist introduces a dual-space hierarchical orchestration system that enables closed-loop refinement of virtual cell models by routing execution discrepancies back to hypothesis or implementation updates, yielding improved benchmark performance with auditable traces.
-
AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery
A survey organizing AI-powered research automation into five workflow stages, defining AutoResearch and Vibe Research, and proposing five evaluation dimensions while noting domain-conditioned limits on autonomy.