REVIEW 5 cited by
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Given that Large Language Models (LLMs) have made significant progress in writing code, can they now be used to autonomously reproduce results from research repositories? Such a capability would be a boon to the research community, helping researchers validate, understand, and extend prior work. To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories. SUPERaims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges (e.g., configuring a trainer), and 602 automatically generated problems for larger-scale development. We introduce various evaluation measures to assess both task success and progress, utilizing gold solutions when available or approximations otherwise. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of this task, and suggests that SUPER can serve as a valuable resource for the community to make and measure progress.
Forward citations
Cited by 5 Pith papers
-
Evaluating LLM Agents on Automated Software Analysis Tasks
A purpose-built, staged LLM agent correctly sets up and executes software analysis tools on 33 of 35 benchmark tasks, outperforming general-purpose agent baselines by at least 17 percentage points.
-
Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks
LLM transcript scanners can surface real validity flaws in agentic benchmarks, including ground-truth leakage and answer-format ambiguity, but sensitivity varies by criterion, benchmark, and model.
-
Throttling Web Agents Using Reasoning Gates
Rebus-based reasoning gates, puzzles built from random word/domain clue sets, impose token costs on LM web agents that are up to 9.2x the generator's cost.
-
AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research
AblationBench is a new benchmark for testing AI planning of ablation experiments, and it shows that current language models recover only a minority of the human reference ablations.
-
CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification
CLOVER is a new Python unit-test-generation benchmark with 845 problems, contexts up to 128k tokens, and coverage-based retrieval, where all 14 evaluated models score below 35% on the hardest task.
Discussion (0). Continue with ORCID to comment.