REVIEW 19 cited by
DABstep: Data Agent Benchmark for Multi-step Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
DABstep: Data Agent Benchmark for Multi-step Reasoning
read the original abstract
We introduce DABstep, a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks. DABstep comprises over 450 real-world challenges derived from a financial analytics platform, requiring models to combine code-based data processing with contextual reasoning over heterogeneous documentation. Each task demands an iterative, multi-step problem-solving approach, testing capabilities in data manipulation, cross-referencing multiple sources, and precise result reporting. The benchmark provides a factoid-style answer format with automatic correctness checks for objective scoring at scale. We evaluate leading LLM-based agents, revealing a substantial performance gap: even the best agent achieves only 14.55% accuracy on the hardest tasks. We detail our benchmark's design, dataset composition, task formulation, evaluation protocol, report baseline results and analyze failure modes. DABstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis.
Forward citations
Cited by 19 Pith papers
-
DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness
A production-grounded, sandbox-graded benchmark of 100 multi-engine data-engineering tasks shows frontier agents top out at 74.9 with no cross-engine winner.
-
LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
LongDS benchmark shows state-of-the-art agents achieve only 48.45% accuracy on long-horizon data analysis tasks, with performance dropping 47 points from early to late turns and state-maintenance errors causing most failures.
-
PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?
PrepBench is a benchmark showing that state-of-the-art LLMs still struggle with natural-language-driven data preparation involving disambiguation, code generation, and workflow translation.
-
DataClawBench: An Agent Benchmark for Exploratory Real-World Financial Data Analysis
Introduces DataClawBench benchmark for exploratory financial data analysis by agents and reports that exploration does not reliably improve task outcomes in noisy cross-domain settings.
-
DataClawBench: An Agent Benchmark for Exploratory Real-World Financial Data Analysis
DataClaw supplies a process-oriented benchmark of real-world noisy data and milestone-annotated tasks that shows seven of eight tested LLMs achieve below 50% accuracy on exploratory analysis.
-
Structure-Grounded Knowledge Retrieval via Code Dependencies for Multi-Step Data Reasoning
SGKR uses function-call dependency graphs to retrieve structured code knowledge, improving LLM correctness on multi-step data reasoning benchmarks over similarity baselines.
-
UrbanDS: A Graph-Guided LLM Multi-Agent System for Data-Intensive Urban Tasks
A graph of reusable dataset skills and relations lets LLM agents discover and join the right urban datasets, outperforming prior data-science agents on data-intensive tasks.
-
Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
A unified corpus of 957k trial outcomes shows frontier progress is uneven and strict all-pass aggregation obscures capability and can reorder agents.
-
Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery
HypoArena is a 988-case benchmark that asks LLMs to generate hypothesis sets from conclusion-free reconstructed contexts and ranks 15 models via pairwise arena judgments.
-
VESTA: Visual Exploration with Statistical Tool Agents
VESTA introduces dynamic tool creation for VLMs that outperforms static-tool and no-tool baselines on distribution fitting, time series, and astronomy tasks in the new DAWN benchmark.
-
DataClawBench: An Agent Benchmark for Exploratory Real-World Financial Data Analysis
DataClawBench is a new benchmark for exploratory real-world financial data analysis that shows increased exploration by LLM agents does not reliably produce task-relevant progress or correct answers.
-
KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving
KAIROS reduces power by 27% on average (up to 39.8%) for agentic AI inference by using long-lived context to jointly manage GPU frequency, concurrency, and request routing across instances.
-
Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows
Finch is a new benchmark with 172 composite workflows and 384 tasks from real enterprise data that shows top AI models like GPT-5.1 Pro pass only 38.4% of workflows under human evaluation.
-
FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data
FDABench is a 2,007-task benchmark for AI agents that must combine structured databases with documents, web, image, video, and audio sources to answer analytical queries.
-
GRACE-DS: a Guarded Reward-guided Agent Correction Environment in Data Science
GRACE-DS supplies metrics and a guarded sandbox for end-to-end evaluation of LLM AutoML agents on organization-specific tabular tasks, with flexible iterative interaction outperforming baselines on hidden-test quality...
-
Unsupervised Skill Discovery for Agentic Data Analysis
DataCOPE uses verifier-guided contrastive distillation from agent trajectories to discover skills, yielding average gains of 9.71% on report-style and 32.30% on reasoning-style data analysis tasks across four model settings.
-
Agentic AI Workload Characteristics
Agentic workloads with context caching become decode-dominated with high KV-cache reuse and show tool use shifting from early read/explore to later execute/write phases.
-
Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media
Presents a new question-based evaluation framework for LLMs on aggregated social media text and reports that performance declines with input scale, task complexity, and numerical operations beyond 500 instances.
-
CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents
A decoupled exploration-selection framework for test-time scaling lifts data-science agent accuracy by up to roughly 12 percentage points over matched baselines.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.