Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

DSAEval claims that data science agents can be reliably evaluated on 641 open-ended, real-world problems from 285 datasets using multimodal observations, multi-query sessions, and LLM judges, and that Claude-Sonnet-4.5 leads the ranking whi

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 09:28 UTC pith:EXDUE3AZ

load-bearing objection DSAEval is a genuinely useful benchmark integration — broad, realistic, multimodal — but its LLM-judge/reference loop is unvalidated and one ARIMA reference is internally inconsistent, so the headline rankings and multimodal gains are conditional until human calibration and error analyses are added. the 4 major comments →

arxiv 2601.13591 v2 pith:EXDUE3AZ submitted 2026-01-20 cs.AI cs.CL

DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems

Maojun Sun , Yifei Xie , Yue Wu , Ruijian Han , Binyan Jiang , Defeng Sun , Yancheng Yuan , Jian Huang This is my paper
classification cs.AI cs.CL
keywords data science agentsLLM-as-a-judgemultimodal perceptionmulti-query interactionagentic benchmarkdata science evaluationmultidimensional scoringopen-ended data analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DSAEval is a benchmark that claims to measure how well AI data science agents handle real-world, open-ended problems: 641 tasks built from 285 datasets, spanning tabular, time-series, text, and image data, with no single standard answer. It tries to establish three things: that such open-ended work can be scored reliably by LLM judges across reasoning, code, and final results; that current agents perform strongly on tabular data and routine workflow steps but much worse on unstructured data and model training; and that letting agents see plots and images, not just text, improves performance by 2.04% to 11.30% on vision-related tasks. If right, DSAEval gives the field a reusable instrument for ranking agents and a measurement of where the frontier actually is: Claude-Sonnet-4.5 leads overall, GPT-5.2 is the most token-efficient, and MiMo-V2-Flash is the cheapest per task.

Core claim

On its own terms, DSAEval is a new evaluation instrument for data science agents. It consists of 641 problems derived from 285 real-world datasets and solved in a sandboxed Jupyter environment with GPU access; each session is a sequence of connected queries on one dataset, and the agent receives observations in text, table, and image form. Each problem carries a soft ground truth generated by GPT-5 and Grok-4 from winning Kaggle notebooks and textbooks, then validated by human experts. Two independent LLM judges, Claude-Haiku-4.5 and GPT-5.1, score the agent's notebook and report on reasoning, code, and results with weights 0.3, 0.3, and 0.4. The paper reports that Claude-Sonnet-4.5 scores h

What carries the argument

The central object is the DSAEval benchmark itself, built from Question-Reasoning-Answer (QRA) triplets that serve as soft ground truth. The scoring mechanism is the weighted judge formula S_total = 0.3 * S_reason + 0.3 * S_code + 0.4 * S_result, applied by two LLM judges. The interaction mechanism is the multimodal observation tuple o_t = {text, table, image} returned by a persistent Jupyter sandbox across a multi-query session. These three pieces together make open-ended, non-exact-match evaluation possible.

Load-bearing premise

The entire ranking depends on LLM-generated reference answers and LLM judges measuring genuine data-science quality rather than stylistic similarity; the paper's own limitations section admits judge randomness, and the Appendix C ARIMA example (train on first 5000, test on indices 1000-1014) shows a reference can be internally inconsistent.

What would settle it

Run a human expert audit on a random sample of DSAEval judge scores, specifically the ARIMA forecasting task in Appendix C; if correcting the internally inconsistent test window changes model scores by more than one point, or if expert agreement with the LLM judges falls below a pre-registered threshold, the rankings and multimodal gains should be treated as unvalidated.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • DSAEval gives a single reusable testbed where 641 real-world tasks can rank agents and track progress, replacing smaller, single-modality, single-step benchmarks.
  • The reported hierarchy implies current data science agents are not yet reliable deep-learning engineers: their weakest stage is model training and optimization and forecasting, while their strength is tabular data ingestion and preparation.
  • The multimodal gains imply that agentic vision-language models should be given image observations in data science environments; withholding plots costs 2% to 11% on vision-related work.
  • The efficiency and cost analysis implies the best-scoring model is not always the best deployment choice when tokens or budget are constrained; GPT-5.2 and MiMo-V2-Flash offer better trade-offs.
  • The planned expansion to 2,000+ datasets and 10,000+ QRA pairs, if released, would support large-scale evaluation and reinforcement learning from execution traces.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The judge chain has a known failure mode that the paper partly acknowledges in its limitations section: if judges reward answers that resemble LLM-generated references rather than check substantive correctness, the rankings and the multimodal gains could be inflated; a human audit of judge scores is the natural next check.
  • Inference: The Appendix C ARIMA task, which says to train on the first 5000 observations but evaluates on indices 1000-1014, would let an agent test on training data; that specific reference appears internally inconsistent, suggesting other synthesized references should be manually re-audited before the benchmark is used as ground truth.
  • Inference: The same multi-query, GPU-backed sandbox format could be reused as a training environment rather than only an evaluation one, since it produces detailed execution traces—code, failures, and plots—that are well suited for fine-tuning agents from execution feedback.
  • Inference: Because the paper finds the largest losses on unstructured data and modeling tasks, a targeted extension would be a diagnostic sub-benchmark isolating vision, NLP, and hyperparameter-tuning sub-tasks to determine whether the gaps come from perception, reasoning, or tool use.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. DSAEval is a benchmark consisting of 641 data science problems derived from 285 real-world datasets and 50 textbooks, with LLM-synthesized Question–Reasoning–Answer (QRA) triples used as soft ground truth. Agents operate in a GPU-equipped sandbox with multimodal observations and multi-query sessions; a weighted LLM judge scores reasoning, code, and final results. The paper evaluates 11 LLM/VLM agents, reporting Claude-Sonnet-4.5 as the overall leader (8.164), GPT-5.2 as the most token-efficient, MiMo-V2-Flash as the most cost-effective, and consistent multimodal perception gains of 2.04–11.30% on vision-related tasks.

Significance. If the evaluation chain is valid, DSAEval is a genuinely useful and unusually broad benchmark: it combines real notebook traces, heterogeneous data modalities, deep learning tasks, multi-query workflows, and multi-dimensional scoring. The public task viewer, detailed execution logs, and large dataset coverage are concrete assets, and the domain/task breakdown gives actionable findings about where current agents fail. However, the headline claims all rest on LLM-generated references and LLM judges, and the manuscript does not provide positive human-calibration evidence for either component. A concrete QRA error in Appendix C shows that the reference/judge loop can penalize a literally correct reading of the stated task. The benchmark concept is sound, but the reported rankings and multimodal gains need additional validation before they can be treated as reliable.

major comments (4)
  1. [§3.1, §3.3, §7.1, Appendix G] The evaluation chain is unvalidated in a load-bearing way. QRA references are synthesized by GPT-5 and Grok-4 (§3.1), and final scores come from two LLM judges (§3.3). §7.1 asserts that the judges 'align closely with human evaluation', but no agreement statistic, sample size, or validation protocol is reported. Appendix G only documents a negative result for an alternative judge (Doubao). Since Eq. (1) feeds every final score through this chain, and S_result has weight 0.4, judge biases or reference errors propagate directly into Figure 1 and Table 2. Please report human–judge agreement, judge–judge agreement, and the fraction of QRA pairs corrected during the human validation mentioned in §3.1. Without this, the rankings and multimodal gains may partly measure conformity to LLM-generated reasoning styles rather than substantive correctness.
  2. [Appendix C (QRA id 508) and Appendix F] The ARIMA reference is internally inconsistent. The question in QRA id 508 says to train on the first 5000 observations and forecast the next 15, but then defines the test window as indices 1000–1014, which lie inside the training window. Appendix F's evaluation of run id 508 penalizes an agent that used test_data.iloc[1000:1015] with Final Results = 2 because its reported metrics differ from the reference, even though the agent followed the literal test-window instruction. This is a concrete, confirmed error in the reference/judge loop, and no audit of the remaining 640 QRA pairs is reported. Because S_result is weighted 0.4 in Eq. (1), such a single bad reference can shift task-level scores by several points and, if nontrivial in frequency, alter aggregate rankings.
  3. [§4.1 and §7.1] The judge model identity is inconsistent and affects reproducibility. §4.1 states that the two judges are Claude-Haiku-4.5 and GPT-5.1, while §7.1 says GPT-5.2 and Claude-Haiku-4.5 were selected as judges. The final scores cannot be reproduced without knowing which GPT-5 model actually produced them. Please correct this and report the exact model versions, prompts, and sampling settings used for the final evaluation.
  4. [Figure 1, Table 2, §5] No error bars, confidence intervals, or significance tests are reported for any headline claim, despite §7.1 acknowledging LLM randomness. The 0.451 gap between Claude-Sonnet-4.5 and GPT-5.2 may be within run-to-run variation, and the Table 2 gains (2.04–11.30%) are based on what appear to be single runs of three models over four domains. Report repeated runs with confidence intervals, or at least a paired significance test, for RQ1 and RQ3. Without this, the claims that one model is 'strongest' and that multimodal perception 'consistently improves' performance are not statistically established.
minor comments (3)
  1. [Abstract vs. §4.1/§5] The abstract reports 13 evaluated models and names MiMo-V2-Pro as the duration leader, but the body evaluates 11 models, includes only MiMo-V2-Flash, and names GPT-5.2 as the efficiency leader. Please align the abstract with the experimental content and correct the model name.
  2. [§2] The text says 'Figure 1 provides a detailed comparison with selected related works,' but Figure 1 is the overall performance chart; the comparison table is Table 1. Please fix the cross-reference.
  3. [Abstract and throughout] There is a typo 'data anlysis' in the abstract, and the spelling of 'MiMo' vs 'Mimo' is inconsistent across the paper. Please standardize.

Circularity Check

0 steps flagged

No load-bearing circularity: DSAEval's benchmark construction is grounded in real notebook outputs and claimed human validation; the LLM-generated soft ground truth / LLM-judge loop is an evaluation-validity risk, not an equation-level circular reduction.

full rationale

Walking the claimed chain: QRA references are synthesized by GPT-5/Grok-4 from real notebooks (Sec. 3.1), agents execute in a sandbox (Sec. 3.2), and GPT-5.1/Claude-Haiku-4.5 score artifacts against those references (Sec. 3.3). If the reference set were purely LLM stylistics and the judge rewarded stylistic agreement, the rankings could be self-referential. But the paper constrains references to real notebook output (Appendix B: 'The answer MUST be real output from the notebook'), instructs the judge to accept alternative valid solutions (Appendix B: 'The [STANDARD_ANSWER] represents just one possible way'), and claims human validation (Sec. 3.1: 'Finally, human experts validated these data'). These are external anchors. The reported scores are empirical measurements, not quantities derived from the reference definitions by construction, so there is no specific Eq.-to-Eq. reduction or fitted-parameter-renamed-as-prediction step. The self-citations (LAMBDA, the authors' survey) are contextual, not load-bearing. The ARIMA QRA id 508 inconsistency (train on first 5000 observations, evaluate indices 1000-1014) and the Sec. 7.1 assertion that judges 'align closely with human evaluation' without reported agreement are real validation deficiencies, but they are correctness risks rather than circularity. The low nonzero score reflects the under-reported external-calibration anchor, not a confirmed circular step.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

No new physical entities are postulated. The main assumptions are that LLM-generated references and LLM judges provide valid correctness signals, and that notebook/lesson selection represents real data science. The single hand-chosen numeric parameter is the evaluation weighting.

free parameters (1)
  • Evaluation weights alpha/beta = alpha=0.3, beta=0.3
    Hand-set in Section 3.3; final total score = 0.3*reasoning + 0.3*code + 0.4*result. Rankings could shift if these weights change.
axioms (4)
  • ad hoc to paper LLM-generated QRA pairs (GPT-5/Grok-4), after human validation, are valid soft ground truth for open-ended data science tasks.
    Section 3.1; no quantitative human-agreement metric is reported.
  • domain assumption LLM-as-judge scores (Claude-Haiku-4.5 and GPT-5.1) correlate with true solution quality.
    Section 3.3 and Appendix G; judges selected on unreported empirical observations of alignment with human evaluation.
  • domain assumption Highest-voted Kaggle notebooks and selected textbooks are representative of real-world data science problems.
    Section 3.1; selection rule intended to ensure quality.
  • domain assumption 20 interaction turns and 1-hour per-iteration timeout are sufficient to complete real workflows.
    Section 4.1; truncation may bias scores for long, iterative deep-learning projects.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems." pith.science (2026). https://pith.science/paper/EXDUE3AZ

@misc{pith2026260113591,
  author       = {Pith},
  title        = {Pith review of: DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EXDUE3AZ}},
  note         = {Machine review of arXiv:2601.13591}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent LLM-based data agents aim to automate data science tasks ranging from data analysis to deep learning. However, the open-ended nature of real-world data science problems, which often span multiple taxonomies and lack standard answers, poses a significant challenge for evaluation. To address this, we introduce DSAEval, a benchmark comprising 641 real-world data science problems grounded in 285 diverse datasets, covering both structured and unstructured data (e.g., image and text). DSAEval incorporates three distinctive features: (1) Multimodal Environment Perception, which enables agents to interpret observations from multiple modalities, including text and vision; (2) Multi-Query Interactions, which mirror the iterative and cumulative nature of real-world data science projects; and (3) Multi-Dimensional Evaluation, which provides a holistic assessment across reasoning, code, and results. We systematically evaluate 13 recent advanced agentic LLMs using DSAEval. Our results show that Claude-Sonnet-4.5 achieves the strongest overall performance, MiMo-V2-Pro and GPT-5.2 lead in duration and step efficiency, respectively, and MiMo-V2-Flash is the most cost-effective. We further demonstrate that multimodal perception consistently improves performance on vision-related tasks, with gains ranging from 2.04\% to 11.30\%. Overall, while current data science agents perform well on structured data and routine data analysis workflows, substantial challenges remain in unstructured domains. Finally, we offer critical insights and outline future research directions.

Figures

Figures reproduced from arXiv: 2601.13591 by Binyan Jiang, Defeng Sun, Jian Huang, Maojun Sun, Ruijian Han, Yancheng Yuan, Yifei Xie, Yue Wu.

Figure 1
Figure 1. Figure 1: Overall performance of all models on DSAEval. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of DSAEval. Left: In the Data Collection Pipeline, raw cases are cleaned and synthesized into Question, Reasoning, and Answer (QRA) pairs using advanced LLMs. Middle: The Data Agent Pipeline orchestrates the agent to solve tasks within a Sandbox Environment. The agent receives multimodal observations and produces a final report and a Jupyter notebook. Right: The Multi-Dimensional Evaluation module… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of the DSAEval benchmark. The suite covers diverse data modalities (left), problem domains [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Fine-grained Performance Analysis. Left: Performance by Domain shows robust capabilities in Data Analysis but significant weaknesses in Computer Vision and NLP. Right: Performance by Task Type highlights proficiency in Data Ingestion but bottlenecks in Statistical Inference and Model Evaluation [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Efficiency and Cost-Effectiveness Analysis. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Scores of Doubao-Seed-1.8 as judge model. [PITH_FULL_IMAGE:figures/full_fig_p034_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Scores of Doubao-Seed-1.8 as judge model. [PITH_FULL_IMAGE:figures/full_fig_p034_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

    cs.AI 2026-06 unverdicted novelty 7.0

    SciAgentArena is a new interactive benchmark for AI agents on scientific tasks that finds agents handle clear data-analysis workflows but struggle with novel insights, self-directed exploration, and open-ended questions.

Reference graph

Works this paper leans on

26 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Data type and domain - Typically, the data type and domain of QRA are the same in each notebook

  2. [2]

    InFindings of the Association for Computa- tional Linguistics: ACL 2025, pages 19796–19821, Vienna, Austria

    Data interpreter: An LLM agent for data sci- ence. InFindings of the Association for Computa- tional Linguistics: ACL 2025, pages 19796–19821, Vienna, Austria. Association for Computational Lin- guistics. Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Ku...

    2025

  3. [3]

    chain of thought

    Reasoning - Must provide a comprehensive, logically connected, and step-by-step explanation... - Do not include or describe code syntax or variable names. - The reasoning should form a clear “chain of thought” from raw data understanding to final result

  4. [4]

    Llms-as-judges: A comprehensive sur- vey on llm-based evaluation methods.Preprint, arXiv:2412.05579. A. Liu and 1 others. 2025. Deepseek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556. MiniMax AI. 2025. Minimax-m2: A model built for max coding & agentic tasks. https://github.com/ MiniMax-AI/MiniMax-M2. Mistral AI....

    Pith/arXiv arXiv 2025

  5. [5]

    modeling

    Llm4ds: Evaluating large language models for data science code generation. OpenAI. 2025. Gpt-5.2 model. https://openai.com/ index/introducing-gpt-5-2/. Shuyin Ouyang, Dong Huang, Jingwen Guo, Zeyu Sun, Qihao Zhu, and Jie M. Zhang. 2025. Dscodebench: A realistic benchmark for data science code genera- tion. Qwen Team. 2025. Qwen3 technical report.Preprint,...

    Pith/arXiv arXiv 2025

  6. [7]

    I have the data mlg-ulb/creditcardfraud...)

    Question - Must be a data science questions in user style with data and target (e.g. I have the data mlg-ulb/creditcardfraud...). - Should align with a clear analytical step such as dataset inspection, preprocessing, visualization... - DO not ask question directly within the notebook. - MUST start from row data, can not ask questions depend on the last step

  7. [9]

    - For data visualization, keeping the image id in the last (e.g

    Answer - The answer MUST be real output from the notebook. - For data visualization, keeping the image id in the last (e.g. <image_id:1>). - For the confidence score. 1 is totally uncertain, 4 is totally certain

  8. [10]

    data_type

    Quality Requirements - Do no mention the notebook hence the student can not see a notebook. - Ensure diversity across task types. - The final output must be valid JSON, parsable without syntax errors. ==================== ### Example [ { "data_type" : "tabular data", "domain" : "exploratory data analysis", "task_type": "data visualization", "language": "P...

  9. [14]

    - Summarize what these results mean in a concise, well-structured text

    Results & Explanation - Report the key numeric results, metrics, or findings explicitly (e.g., final metrics, key statistics). - Summarize what these results mean in a concise, well-structured text. - If you generated any important artifacts (e.g., figures, model files), mention their filenames and what they contain. - Do NOT provide or reference any sepa...

  10. [15]

    Task Understanding Briefly restate the problem and the main goal in 2–5 sentences

  11. [16]

    Explain how you use the data, which files you select, and any important modeling or analysis choices

    Approach Summary Describe your solution strategy in clear, high-level steps (conceptual only, not low-level inner thoughts). Explain how you use the data, which files you select, and any important modeling or analysis choices

  12. [17]

    Key Implementation Provide some piece of key python codes of your problem solving process with explanations

  13. [18]

    - Summarize what these results mean in a concise, well-structured text

    Results & Explanation - Report the key numeric results, metrics, or findings explicitly (e.g., final metrics, key statistics). - Summarize what these results mean in a concise, well-structured text. - If you generated any important artifacts (e.g., figures, model files), mention their filenames and what they contain. - Do NOT provide or reference any sepa...

  14. [19]

    Core Objective: Did the model answer the[PROBLEM]correctly?

  15. [20]

    The[predicted_reasoning_answer]may adopt a different but equally valid approach

    The[STANDARD_ANSWER]represents just one possible way. The[predicted_reasoning_answer]may adopt a different but equally valid approach

  16. [21]

    Reference Figures

    Visuals: You may receive "Reference Figures" and "Predicted Figures". - DO NOT PENALIZE if the predicted answer has NO figures, *UNLESS* the[PROBLEM]explicitly asks to "plot", "visualize", or "graph". - If both have figures, compare them to ensure the predicted figure shows correct data distribution, trends and insights

  17. [22]

    Analysis

    Code-Answer Consistency: You must verify if the final answer in[predicted_reasoning_answer] is actually derived from[PREDICTED_CODE]. ### Evaluation Dimensions: * Consistency (true or false): Focus on the alignment between[PREDICTED_CODE]and [predicted_reasoning_answer]. - true: The narrative accurately reflects the code’s final output. - false: The narra...

    2025

  18. [26]

    Specifically, the objectives were: • Clean theText field (lowercasing, punctuation/URL/non-ASCII removal, stopword removal, stemming)

    Task Understanding The task required comprehensive text cleaning and sentiment analysis on the social media dataset. Specifically, the objectives were: • Clean theText field (lowercasing, punctuation/URL/non-ASCII removal, stopword removal, stemming). • Compute V ADER compound sentiment scores and map them to labels: Positive (≥0.05 ), Neutral (−0.05to0.0...

  19. [27]

    Approach Summary I loaded the processed dataset (732 posts) and implemented a cleaning pipeline using NLTK. The pipeline applies six sequential transformations: lowercasing, regex-based URL removal, non- ASCII removal (emojis), punctuation stripping, stopword filtering, and Porter stemming. V ADER analysis was applied to theoriginaltext to preserve contex...

  20. [28]

    ,→Lowercase 4text = re.sub(r'http\S+|www\S+|https\S+',' ', text) # 2

    Key Implementation 1# Text Cleaning & VADER Pipeline 2def clean_text(text): 3text = text.lower() # 1. ,→Lowercase 4text = re.sub(r'http\S+|www\S+|https\S+',' ', text) # 2. Remove ,→URLs 5text = text.encode('ascii','ignore').decode('ascii') # 3. Remove ,→non-ASCII 6text = text.translate(str.maketrans(' ',' ', string.punctuation)) # 4. ,→Remove Punctuation ...

  21. [29]

    enjoy beauti day

    Results & Explanation First 5 Clean Text Samples with V ADER Scores: Row Original Text Cleaned Text Score V ADER Original 1 Enjoying a beautiful day... enjoy beauti day... 0.8221 Positive Positive 2 Traffic was terrible... traffic terribl morn -0.4767 Negative Negative 3 Just finished an amaz... finish amaz workout 0.6239 Positive Positive 4 Excited about...

  22. [103]

    Removing non-ASCII characters

  23. [114]

    Removing punctuation

  24. [136]

    "" 15if pd.isna(text): 16return

    Stemming 14""" 15if pd.isna(text): 16return "" 17 18# 1. Lowercase 19text = text.lower() 20 21# 2. Remove URLs 22text = re.sub(r'http\S+|www\S+|https\S+','', text, flags=re. ,→MULTILINE) 23 24# 3. Remove non-ASCII characters (emojis, special characters) 25text = text.encode('ascii','ignore').decode('ascii') 26 27# 4. Remove punctuation 28text = text.trans...

  25. [2024]

    InProceedings of the 41st Interna- tional Conference on Machine Learning, ICML’24

    Infiagent-dabench: evaluating agents on data analysis tasks. InProceedings of the 41st Interna- tional Conference on Machine Learning, ICML’24. JMLR.org. Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. 2024a. Mlagentbench: evaluating language agents on machine learning experimentation. InProceed- ings of the 41st International Conference on Machin...

    2024

  26. [2025]

    Google DeepMind

    Dabstep: Data agent benchmark for multi-step reasoning. Google DeepMind. 2025. Gemini 3 pro model card. https://deepmind.google/models/gemini/ pro/. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A su...

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.