REVIEW 8 cited by
DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have demonstrated impressive language/vision reasoning abilities, igniting the recent trend of building agents for targeted applications such as shopping assistants or AI software engineers. Recently, many data science benchmarks have been proposed to investigate their performance in the data science domain. However, existing data science benchmarks still fall short when compared to real-world data science applications due to their simplified settings. To bridge this gap, we introduce DSBench, a comprehensive benchmark designed to evaluate data science agents with realistic tasks. This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions. DSBench offers a realistic setting by encompassing long contexts, multimodal task backgrounds, reasoning with large data files and multi-table structures, and performing end-to-end data modeling tasks. Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG). These findings underscore the need for further advancements in developing more practical, intelligent, and autonomous data science agents.
Forward citations
Cited by 8 Pith papers
-
CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
No evaluated frontier LLM agent shows reliable causal thinking on CausalGame’s 14 SCM-based drone games; survival and mechanistic understanding remain largely decoupled.
-
ExplainBench: Evaluating Code Explanations from Agents
Agent explanation quality can be scored automatically with LLM questionnaires, and this score ranks agents differently from SWE-bench Verified.
-
Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery
HypoArena is a 988-case benchmark that asks LLMs to generate hypothesis sets from conclusion-free reconstructed contexts and ranks 15 models via pairwise arena judgments.
-
StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
StatEval is a new 16,000-question statistics benchmark showing that even strong LLMs score below 60% on research-level statistical proof tasks.
-
Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering
Matryoshka Agent’s orchestrator–sub-agent hierarchy plus tree-ranked RL raises MLE-Dojo HumanRank, letting a 4B orchestrator approach o4-mini and giving a 30B coder up to 36.7% relative gain.
-
CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents
A decoupled exploration-selection framework for test-time scaling lifts data-science agent accuracy by up to roughly 12 percentage points over matched baselines.
-
How Far Are AI Scientists from Changing the World?
This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.
-
A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications
A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.
Discussion (0). Sign in to comment.