Pith. sign in

REVIEW 3 cited by

ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.00964 v3 pith:T7G7NS3R submitted 2025-02-03 cs.SE cs.AI

Harshith Padigela , Chintan Shah , Dinkar Juyal This is my paper · ORCID
classification cs.SEcs.AI
keywords ml-dev-benchdevelopmentagentsbenchmarktasksexistinggithubhandling
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this report, we present ML-Dev-Bench, a benchmark aimed at testing agentic capabilities on applied Machine Learning development tasks. While existing benchmarks focus on isolated coding tasks or Kaggle-style competitions, ML-Dev-Bench tests agents' ability to handle the full complexity of ML development workflows. The benchmark assesses performance across critical aspects including dataset handling, model training, improving existing models, debugging, and API integration with popular ML tools. We evaluate three agents - ReAct, Openhands, and AIDE - on a diverse set of 30 tasks, providing insights into their strengths and limitations in handling practical ML development challenges. We open source the benchmark for the benefit of the community at \href{https://github.com/ml-dev-bench/ml-dev-bench}{https://github.com/ml-dev-bench/ml-dev-bench}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework

    cs.DC 2026-07 conditional novelty 7.0 of 10

    Agent serving faces non-inference bottlenecks (up to 48% of latency), a 4.4x serving-capacity loss from long context, and 4.9x cost amplification from snapshot-based sandbox suspension.

  2. How Far Are AI Scientists from Changing the World?

    cs.AI 2025-07 conditional novelty 4.0 of 10

    This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.

  3. AI Scientists Fail Without Strong Implementation Capability

    cs.AI 2025-06 conditional novelty 4.0 of 10

    AI scientist systems can propose ideas but cannot reliably implement and verify experiments, making the implementation gap, not idea generation, the current bottleneck.

Pith tools