AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

An Luo; Ashish Kundu; Charles Fleming; Fangqiao Tian; Galin Jones; Ganghua Wang; Jayanth Srinivasa; Jie Ding; Jin Du; Mingyi Hong

REVIEW 3 major objections 2 minor 1 cited by

AI agents lag top humans on 17 domain-specific data science challenges; strongest results come from human-AI teams.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 22:15 UTC pith:ACAXJY2D

load-bearing objection Useful multi-industry agent benchmark and competition; the headline claim that AI-only agents lag top human-AI teams is plausible but uncheckable from the abstract alone. the 3 major comments →

arxiv 2603.19005 v3 pith:ACAXJY2D submitted 2026-03-19 cs.LG cs.AIstat.ME

AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

An Luo , Jin Du , Xun Xian , Robert Specht , Fangqiao Tian , Ganghua Wang , Xuan Bi , Charles Fleming

show 7 more authors

Ashish Kundu Jayanth Srinivasa Mingyi Hong Rui Zhang Tianxi Li Galin Jones Jie Ding

This is my paper

classification cs.LG cs.AIstat.ME

keywords AgentDSdomain-specific data sciencehuman-AI collaborationAI agentsbenchmarkLLM agentscompetition

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces AgentDS, a benchmark and open competition that tests how far current AI agents can go on real domain-specific data science work, and where humans still matter. Seventeen challenges span six industries—commerce, food production, healthcare, insurance, manufacturing, and retail banking. Twenty-nine teams and eighty participants competed; their results were compared against AI-only baselines. The central finding is that pure AI agents fall short of the top quartile of human-AI competitors: domain-specific reasoning remains hard for today's agents, while the best outcomes come from people and models working together. The work challenges claims of full automation and frames human expertise as still essential, while pointing to the kinds of reasoning that next-generation agents will need to close the gap.

Core claim

On the 17 AgentDS challenges, current AI agents struggle with domain-specific reasoning: AI-only baselines score below the top quartile of competition participants, and the strongest solutions arise from human-AI collaboration rather than full automation.

What carries the argument

The AgentDS benchmark itself: 17 industry-grounded challenges plus an open competition that lets AI-only baselines be ranked against human-AI teams under the same scoring rules.

Load-bearing premise

That the 17 competition tasks and the chosen AI-only baselines fairly represent both real domain-specific data-science difficulty and the current capability frontier of AI agents.

What would settle it

Re-run the same 17 challenges with stronger or differently configured AI-only agents (or a larger, independently scored human cohort) and check whether pure AI then reaches or exceeds the top quartile of human-AI teams.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful multi-industry agent benchmark and competition; the headline claim that AI-only agents lag top human-AI teams is plausible but uncheckable from the abstract alone.

read the letter

The short version: AgentDS is a new multi-industry data-science agent benchmark plus an open competition (17 tasks, 29 teams, 80 people). The abstract’s headline result—AI-only baselines below the top quartile, strongest solutions from human-AI teams—is the kind of finding people will quote. We only have the abstract, so we cannot verify whether the tasks, scoring, and baselines actually support it.

What is new and useful is the resource itself. Six industries (commerce, food production, healthcare, insurance, manufacturing, retail banking), seventeen challenges, open datasets on Hugging Face, and a real competition. Domain-specific agent evals and Kaggle-style agent contests already exist; a multi-industry suite with an explicit human-AI vs AI-only comparison is still a legitimate addition. If the full paper ships task definitions, leaderboards, baseline scaffolding, and scoring rubrics, that is real value for people building or evaluating agents for applied data science.

The soft spot is load-bearing and matches the stress-test note. The claim that current agents struggle with domain-specific reasoning only holds if (1) the challenges actually require domain reasoning rather than generic coding/ML, (2) the AI-only baselines are competitive with the current agent frontier rather than under-tooled strawmen, and (3) scoring does not quietly reward human priors or post-hoc interpretation agents never get. The abstract asserts the ranking and points to a website; it gives no task specs, baseline list, metrics, error bars, or statistical comparison. Circularity risk is low—this is a competition outcome, not fitted math folded back into predictions—but design-bias risk is real until those details are public. I am not manufacturing a flaw; I am saying the central empirical claim is currently uncheckable.

Who this is for: researchers and practitioners in agent evaluation and human-AI tooling for data science. Not a theory paper. I would send it to peer review rather than desk-reject; a serious referee needs to pressure-test baseline selection and scoring design. I would not cite the automation-critique claim until I can inspect the methods. Bring it to reading group only if someone has the full paper and the HF artifacts in hand.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces AgentDS, a benchmark and open competition for domain-specific data science, comprising 17 challenges across six industries (commerce, food production, healthcare, insurance, manufacturing, and retail banking). It reports a competition with 29 teams and 80 participants and compares human–AI collaborative solutions against AI-only baselines. From the abstract, the central claim is that current AI agents struggle with domain-specific reasoning: AI-only baselines fall below the top quartile of competition participants, while the strongest solutions arise from human–AI collaboration, challenging full-automation narratives and highlighting enduring human expertise.

Significance. If the competition design, baselines, and scoring are sound, AgentDS would be a useful multi-industry empirical contribution to agent evaluation and human–AI collaboration research. Open competition results, open-sourced datasets (Hugging Face), and an explicit human–AI vs AI-only comparison are strengths that could ground claims about where agents still fail on domain-specific data science. The work is timely given rapid agent automation claims; its value hinges on whether the 17 tasks and baselines fairly represent real domain difficulty and the current agent frontier.

major comments (3)

The headline result—that AI-only baselines perform below the top quartile of 29 teams while strongest solutions are human–AI—is load-bearing for the paper’s central claim. The abstract asserts this comparison but does not specify the AI-only baseline models, agent scaffolding/tools, prompting or planning setup, or whether baselines had access to the same tools and compute as participants. Without those details (and without full-text verification), it is impossible to judge whether underperformance reflects a genuine capability gap or under-tuned/under-tooled baselines relative to the agent frontier participants could use.
The claim that agents ‘struggle with domain-specific reasoning’ requires that the 17 challenges actually isolate domain-specific reasoning rather than generic coding, EDA, or standard ML. The abstract lists industries but supplies no task definitions, required domain priors, data schemas, or scoring rubrics. If scoring systematically rewards human-provided domain knowledge or post-hoc interpretation unavailable to pure agents, the human–AI advantage is partly by construction. This design choice must be documented and stress-tested for the underperformance claim to hold.
No statistical comparison, error bars, significance tests, or ablation of human vs. tool contributions appear in the abstract. A quartile ranking over 29 teams is not, by itself, a robust demonstration of a capability gap. The manuscript needs transparent metrics, variance across tasks/industries, and an analysis separating human domain input from agent execution; otherwise the cross-team ranking cannot support the strong narrative claim against complete automation.

minor comments (2)

Abstract-only review: figure/table clarity, notation, related-work coverage, and reproducibility appendix cannot be assessed without the full text. Once the full manuscript is available, standard presentation checks (task cards, baseline tables, metric definitions, leaderboard protocol) will be needed.
The abstract would benefit from naming the AI-only baseline systems and the primary evaluation metric(s) so readers can immediately gauge the strength of the comparison.

Circularity Check

0 steps flagged

No definitional or construction circularity: AgentDS reports empirical competition outcomes, not a derivation that folds inputs into claimed predictions.

full rationale

This is an abstract-only review of a benchmark-and-competition paper. The central claims are empirical: 17 domain challenges, 29 teams / 80 participants, AI-only baselines below the top quartile of participants, and strongest solutions from human-AI collaboration. There is no mathematical derivation chain, no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, and no ansatz smuggled in via self-citation. The result is the competition outcome itself, not a quantity forced by construction from the paper's own inputs. Design or selection risk (whether tasks and baselines fairly represent the frontier) is a validity concern, not circularity under the enumerated patterns. With only the abstract available, no quoteable reduction of the form Eq. X = Eq. Y by construction exists. Score 0 is therefore the correct honest finding.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

Abstract-only review. No free parameters are fitted in a mathematical sense; the work is an empirical competition. Core unstated premises are that the 17 tasks are representative of domain-specific data science, that the AI-only baselines are competitive with the current agent frontier, and that competition rankings fairly measure capability rather than tool access or time budgets. No new physical entities are invented.

axioms (3)

domain assumption The 17 AgentDS challenges are representative of real domain-specific data science difficulty across the six named industries.
Central comparative claim depends on task validity; abstract asserts domain-specificity without providing task construction criteria.
domain assumption The chosen AI-only baselines fairly represent current agent capability on these tasks.
Ranking of AI-only vs. human-AI teams is only meaningful if baselines are not deliberately weak or outdated.
domain assumption Competition scoring and participation conditions allow fair comparison between pure AI runs and human-AI teams.
Abstract does not detail time limits, tool access, or whether humans could use stronger models than the baselines.

pith-pipeline@v1.1.0-grok45 · 6191 in / 2403 out tokens · 21174 ms · 2026-07-13T22:15:46.547484+00:00 · methodology

0 comments

read the original abstract

Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform below the top quartile of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Nonuniformity Principle in Human-AI Coworking
cs.AI 2026-07 conditional novelty 6.0

Optimal oversight schedules in multi-step AI workflows place human checkpoints early and with non-decreasing gaps between them.