Pith. sign in

REVIEW 3 major objections 5 minor 47 references

Even the best AI agent scores only 74.9 on real industrial data-engineering tasks, and no model wins across engines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 19:51 UTC pith:OWNZSBWA

load-bearing objection Solid industrial benchmark paper: the “far from solved” ceiling is credible; the “strict engine specialization” half is thinner than the abstract sells. the 3 major comments →

arxiv 2607.28033 v1 pith:OWNZSBWA submitted 2026-07-30 cs.AI

DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness

Debin Meng , Jiaming Yang , Zefang Zong , Tengyue Xu , Haining Xie , Yang Li , Peng Chen This is my paper
classification cs.AI
keywords Data EngineeringAutonomous AgentEnterprise Data SystemsBenchmarkMulti-engine SQLRule-based evaluationETLStreaming
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DataClawEval is a new benchmark that asks whether autonomous AI agents can finish real enterprise data-engineering jobs end to end—not just write a SQL query or a short analysis script. It builds 100 tasks from production code across five engines (PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL), covering batch and streaming work in domains like ops, growth analytics, security, and ads. Each agent must explore live tables in an isolated sandbox, write and debug code, and materialize correct outputs; scoring uses case-specific deterministic scripts rather than an LLM judge. Across 16 frontier models the top overall score is only 74.9, HiveSQL is hardest, MySQL is easiest, and different models lead different engines. The paper’s point is that full-stack data engineering remains an open, engine-specialized challenge, and that execution-grounded grading is required to measure it honestly.

Core claim

On 100 production-derived, sandbox-executed data-engineering tasks spanning five engines, the strongest of 16 frontier agents reaches only 74.9 overall; no single model leads every engine, engine difficulty is highly uneven, and token spend does not track quality—so autonomous end-to-end data engineering is still unsolved and models show strict domain specialization rather than general proficiency.

What carries the argument

DataClawEval itself: a human-in-the-loop pipeline that turns desensitized production code into answer-identifiable tasks (LLM-reconstructed intents and inputs, expert perturbations for discriminability, case-specific graders), then scores agents in fresh Docker sandboxes with a weighted mix of artifact correctness and process quality under deterministic rule-based scripts.

Load-bearing premise

That tasks rebuilt from one enterprise’s cleaned production code, plus synthetic inputs and a fixed agent harness, fairly stand in for industrial data engineering in general so the 74.9 ceiling and engine specialization will hold elsewhere.

What would settle it

Re-run the same 16 models on a second, independently sourced suite of production data-engineering tasks (different company or public multi-engine corpus) with the same sandbox-and-rule protocol; if several models clear ~90 overall or one model leads all five engines, the claimed open-challenge ceiling and specialization thesis fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Leaderboards that only test Text-to-SQL or final-answer analysis will overstate readiness for production ETL and streaming jobs.
  • Progress should be reported per engine (especially HiveSQL and FlinkSQL), not only as one average score.
  • Case-specific rule-based graders that execute outputs against live engines become the standard for this domain; generic LLM judges are shown to inflate and destabilize scores.
  • Released tasks, containers, and graders give a shared testbed for measuring whether future agents close the gap without changing the harness mid-comparison.
  • Tool-call volume and token spend are poor proxies for quality; efficient exploration matters more than retry thrash.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Teams deploying a single ‘data agent’ may need engine-specialized models or routers rather than one generalist until cross-engine transfer improves.
  • The large process-score gap under LLM judges suggests trajectory logging and post-run verification will become first-class training signals, not just product metrics.
  • Bilingual and timeout analyses hint that harness limits and prompt language can quietly reorder rankings; future suites may need parallel translations and timeout-robust scoring.
  • If differential testing with expert perturbations is what makes tasks answer-identifiable, similar construction could transfer to neighboring ops domains (infra-as-code, ML pipeline debugging).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. DataClawEval introduces a 100-task executable benchmark for autonomous data-engineering agents, derived from production code and spanning PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL. Tasks are reconstructed via a human-in-the-loop pipeline (intent/table synthesis, differential perturbation checks, case-specific graders) and scored in isolated Docker sandboxes by deterministic rule-based scripts that combine artifact correctness with process metrics (exploration, efficiency, self-verification; α≈0.7). Under a fixed Tencent CodeBuddy harness, 16 frontier LLMs are evaluated once per task. The strongest model reaches only 74.9 overall; MySQL is easiest and HiveSQL hardest; engine leaders differ; token/tool-call volume does not track quality. Ablations argue that LLM-as-judge scoring is inflated and unstable relative to rule-based ground truth, and multi-run checks on four models show non-trivial score and pass-rate variance. The suite, containers, and graders are released.

Significance. If the empirical picture holds, the paper supplies the first production-grounded, multi-engine, end-to-end harness for data-engineering agents and a clear negative result: current frontier agents are far from reliable industrial ETL/stream engineering under live execution. Strengths that should be credited explicitly include (i) case-specific deterministic graders and containerized environments rather than LLM-as-judge, (ii) differential-testing style construction to make tasks answer-identifiable, (iii) joint artifact+process scoring, (iv) a controlled 16-model comparison under one scaffold, and (v) full public release of tasks, sandboxes, and grade.py scripts. These make the benchmark immediately usable and the “unsolved” claim falsifiable by future systems.

major comments (3)
  1. [Abstract; §4.1–4.2; Table 2; §5.2; Appendix C] Abstract and §4.2 treat “no single model dominates, as each excels on a different engine” as a co-equal half of the central claim with the 74.9 ceiling. Table 2 engine crowns (e.g., Claude Opus 4.8 on PySpark 83.8, DeepSeek V4 Flash on FlinkSQL 85.0) are single-run means over small per-engine n (12–28 tasks). The paper’s own §5.2 multi-run study on four agents shows max@3−min@3 gaps of 12.6–26.8 points and pass@3−pass^3 gaps of 8–26%; Appendix C shows timeout rates up to 22% that deflate means by up to 13.3 points and can reorder models (GLM 5.2). The overall “far from solved” band (~60–75) is robust; the strict specialization narrative is not yet secured. Either report multi-run engine means (or bootstrap CIs) for all 16 models, or qualify the specialization claim to match the single-run evidence.
  2. [§4.1; Fig. 4; Appendix C] §4.1 fixes one agent scaffold (Tencent CodeBuddy) and a wall-clock limit for all models. Engine-specific rankings and tool-call efficiency (Fig. 4) therefore confound intrinsic model skill with scaffold/tooling fit and timeout policy. Appendix C already shows timeouts can reorder the leaderboard. The manuscript should state this confound explicitly when interpreting engine winners and, where feasible, report completed-run means alongside full-run means, or a short sensitivity check under a second harness/time budget for a subset of engines.
  3. [§3.1; Fig. 1; §7] External validity of the 74.9 ceiling and engine difficulty ordering rests on tasks reconstructed from one enterprise’s desensitized production code, with LLM-inferred intents and synthetic input tables (§3.1 Stages 2–5). Differential expert perturbations improve discriminability within this corpus, but do not establish that difficulty and dialect mix represent industrial data engineering in general. A short limitations paragraph should bound generalization (single-org provenance, synthetic tables, fixed business-domain mix in Fig. 1) so the interpretive leap “thus autonomous data engineering remains unresolved” is scoped to this harness rather than asserted universally.
minor comments (5)
  1. [§3.2 Eq. (1); Appendix F] Eq. (1) and the surrounding text set α=0.7 “in most” cases, while Appendix F case studies use α∈{0.5,0.6,0.7}. State the distribution of α across the 100 tasks and whether overall scores are sensitive to a global α sweep.
  2. [Table 1; §1] Table 1 lists “DataClawBench” and “Ours (DataClawEval)” with similar names; a one-sentence disambiguation in §1 or the table caption would reduce confusion with the related-work baseline.
  3. [§3.2; Appendix F] Process sub-weights (exploration 35 / efficiency 40 / self-verification 25 in Appendix F) are free parameters not justified in the main text. Briefly motivate or note they are fixed a priori.
  4. [Figure 1; Appendix A Table 4] Figure 1 percentages and engine counts (e.g., PrestoSQL 12%) should be checked against Table 4’s full listing for consistency in the camera-ready.
  5. [Title page; §3.1] Typos/consistency: “Sun Yat-Sun University” on the author block; “answer-identifiable” is used well but could be defined once at first use in §3.1.

Circularity Check

0 steps flagged

No circular derivation: empirical benchmark scores are measured outputs, not predictions forced by fitted inputs or self-citation.

full rationale

DataClawEval is a benchmark-construction and multi-model evaluation paper, not a first-principles derivation. The load-bearing claim (strongest of 16 agents scores 74.9; no model dominates all five engines; end-to-end data engineering remains open) is obtained by running external frontier models in a fixed harness on 100 sandbox tasks and scoring materialized artifacts with case-specific deterministic graders (§3.2, §4, Table 2). Task intents, synthetic tables, and graders are author-built via a human-in-the-loop pipeline (§3.1), and the overall score formula Score=α S_artifact+(1−α) S_process with α≈0.7 is an explicit scoring convention—not a fitted parameter later relabeled as a prediction, nor a uniqueness theorem imported from the authors’ prior work. Related-work citations (BIRD, Spider 2.0, DA-Code, DataClawBench, etc.) situate the gap; they do not force the numerical results. LLM-as-judge comparison (§5.1) and stability/tool-call analyses (§5.2–5.3) are diagnostic side studies, not circular proofs of the ceiling. No step reduces Eq./claim X to input Y by construction. Honest finding: no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

As an empirical systems/benchmark paper, load-bearing commitments are methodological: production code as ground truth, LLM+expert task reconstruction, differential discriminability, case-specific graders, fixed agent harness, and the weighted Score=α S_artifact+(1-α) S_process formula. No physical constants or fitted scientific laws; free parameters are scoring and protocol knobs that affect reported ranks.

free parameters (4)
  • alpha (artifact vs process weight) = 0.7 (default)
    Overall score uses α S_artifact + (1-α) S_process; paper sets α=0.7 in most cases and allows task-level changes (§3.2 Eq. 1), directly shaping the 74.9 headline.
  • Process sub-weights (exploration / efficiency / self-verification) = typically 35% / 40% / 25% of process
    Case studies quote splits such as 35/40/25 of the process share (App. F); these hand-set weights affect process scores and thus overall ranks.
  • Low-score threshold and pass criteria in analyses = 50 / 100 for low-score
    Secondary analyses treat score <50 as low-score and define pass@3 / pass^3 (§5.2, App. B); thresholds are analytical choices, not data-fit constants, but they structure stability and error narratives.
  • Per-task wall-clock time limit = predefined per task (numeric limit not stated in main text)
    Harness timeouts zero 4.9% of runs and can deflate means (e.g., GLM 5.2 by 13.3 pts, App. C); the limit value is a protocol parameter.
axioms (5)
  • domain assumption Production implementations retained after desensitization, dedup, and stratified sampling are valid ground-truth solutions for reconstructed intents.
    Stage 1–2 treat cleaned enterprise code as GT against which synthetic inputs and graders are validated (§3.1).
  • domain assumption If expert-perturbed but still-executable code yields different outputs on synthesized inputs, those inputs sufficiently identify the intended semantics (differential testing).
    Stage 3 uses McKeeman-style differential testing as the discriminability criterion (§3.1).
  • domain assumption Case-specific rule-based graders that full-score GT and not-full-score perturbations are an objective measure of artifact and process quality.
    Stages 4–5 and §3.2 define success via customized scripts rather than code match or LLM judges.
  • domain assumption Holding the Tencent CodeBuddy scaffold fixed and varying only the LLM isolates model capability differences.
    Explicit experimental control in §4.1; performance may not transfer to other agent implementations.
  • standard math Standard definitions and behaviors of PySpark, HiveSQL, MySQL, Presto/Trino, and FlinkSQL engines in the containers match the semantics agents must satisfy.
    Evaluation executes against live engines in Docker; correctness is engine semantics plus business rubrics (§3.2).
invented entities (2)
  • DataClawEval benchmark suite (100 tasks, sandboxes, grade.py per case) independent evidence
    purpose: Provide an answer-identifiable, multi-engine testbed for end-to-end data engineering agents.
    New artifact bundle constructed via the five-stage pipeline; not a physical entity but the paper’s primary invented evaluation object.
  • Combined artifact+process score with exploration/efficiency/self-verification process axes no independent evidence
    purpose: Grade both materialized outputs and engineering trajectory quality.
    Author-defined composite metric (§3.2); falsifiable only insofar as graders and logs are public—interpretation of ‘good process’ is stipulative.

pith-pipeline@v1.2.0-daily-grok45 · 28096 in / 3763 out tokens · 78696 ms · 2026-07-31T19:51:29.030684+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) and LLM-based agents are increasingly being deployed to automate complex workflows, promising to revolutionize data management and processing. However, existing benchmarks predominantly focus on simplified Text-to-SQL translation or data analysis, leaving the critical and complex domain of end-to-end data engineering largely unexplored. To bridge this gap, we introduce DataClawEval, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios. Built upon production-grade code authored by professional enterprise data engineers, it comprises 100 rigorous, end-to-end tasks spanning five execution engines: PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL. Rather than non-deterministic LLM-as-a-judge scoring, each task is executed within a case-specific, isolated sandbox and graded by deterministic, rule-based scripts. Evaluating 16 frontier agents exposes critical limitations: The strongest model attains only 74.9 overall, and no single model dominates, as each excels on a different engine, revealing strict domain specialization rather than omnipotent proficiency. Thus, autonomous data engineering remains a formidable, unresolved challenge. We release our dataset, containerized environments, and deterministic evaluation scripts at https://github.com/Dicemy/DataClawEval/tree/master

Figures

Figures reproduced from arXiv: 2607.28033 by Debin Meng, Haining Xie, Jiaming Yang, Peng Chen, Tengyue Xu, Yang Li, Zefang Zong.

Figure 1
Figure 1. Figure 1: Composition of the 100-task DataClawEval suite: [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of DataClawEval’s two coupled pipelines. Top (Dataset Construction): real data engineering code is cleaned [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Run-to-run stability of four representative agents [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Average tool calls per task (x-axis) vs. average over [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of low scores across the 100 DataClawE￾val tasks. The 𝑥-axis is the number of the 16 models that score below 50 on the 0–100 scale for a task; the 𝑦-axis counts tasks, stacked by execution engine. Most tasks are rarely scored low: 37 receive no low scores at all, while about a dozen hard tasks, mostly in HiveSQL and PrestoSQL/Trino, account for roughly half of all low-score runs. We examine th… view at source ↗
Figure 6
Figure 6. Figure 6: Characterization of the 79 runs terminated at the harness wall-clock limit. (a) Per-model distribution of the 100 run scores (box: healthy-run IQR and median; red ×: timed-out runs). Timed-out runs affect every model except GPT 5.5, with incidence spanning 0–22%. (b) Mean-score deflation attributable to timed-out runs (mean over healthy runs minus the reported mean), sorted; bar-end labels additionally sho… view at source ↗
Figure 7
Figure 7. Figure 7: Per-case LLM-judge score versus Rule-Based (our) [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    2025.Claude Code: An Agentic Coding Tool

    Anthropic. 2025.Claude Code: An Agentic Coding Tool. Anthropic, PBC. https: //www.anthropic.com/claude-code

    2025

  2. [2]

    Anthropic. 2026. Introducing Claude Opus 4.8. https://www.anthropic.com/ news/claude-opus-4-8

    2026

  3. [3]

    Anthropic. 2026. Introducing Claude Sonnet 5. https://www.anthropic.com/ news/claude-sonnet-5

    2026

  4. [4]

    Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache Flink: Stream and Batch Processing in a Single Engine.Bulletin of the IEEE Computer Society Technical Committee on Data Engineering36, 4 (2015). https://asterios.katsifodimos.com/assets/publications/ flink-debull.pdf

    2015

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv doi:10.48550/arxiv.2107.03374 2021

  6. [6]

    DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Con- text Models.arXiv preprint(2026). arXiv:2606.19348 doi:10.48550/arXiv.2606. 19348

    doi:10.48550/arxiv.2606.19348 2026

  7. [7]

    Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding, Ziyu Liu, Yang JingYi, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang, Yubo Ma, Haodong Duan, Jing Shao, Jiaqi Wang, Dahua Lin, Kai Chen, and Yuhang Zang. 2026. WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation. arXiv preprint(2026). arXiv:2605.10912 doi:10.48550/arXiv.2605.10912

    Pith/arXiv arXiv doi:10.48550/arxiv.2605.10912 2026

  8. [8]

    Kingma, Leandro von Werra, and Thomas Wolf

    Alexander David Egg, Martin Iglesias Goyanes, Andreu Mora, Friso H. Kingma, Leandro von Werra, and Thomas Wolf. 2025. DABstep: Data Agent Benchmark for Multi-step Reasoning.arXiv preprint(2025). arXiv:2506.23719 doi:10.48550/ arXiv.2506.23719

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.23719 2025

  9. [9]

    GLM Team. 2026. GLM-5: From Vibe Coding to Agentic Engineering.arXiv preprint(2026). arXiv:2602.15763 doi:10.48550/arXiv.2602.15763

    Pith/arXiv arXiv doi:10.48550/arxiv.2602.15763 2026

  10. [10]

    GLM Team. 2026. GLM-5V-Turbo: Toward a Native Foundation Model for Multi- modal Agents.arXiv preprint(2026). arXiv:2604.26752 doi:10.48550/arXiv.2604. 26752

    Pith/arXiv arXiv doi:10.48550/arxiv.2604.26752 2026

  11. [11]

    Google DeepMind. 2026. Gemini 3.1 Pro. https://deepmind.google/models/ gemini/pro/

    2026

  12. [12]

    Google DeepMind. 2026. Gemini 3.5: Frontier Intelligence with Ac- tion. https://blog.google/innovation-and-ai/models-and-research/gemini- models/gemini-3-5/

    2026

  13. [13]

    Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. 2024. InfiAgent-DABench: Eval- uating Agents on Data Analysis Tasks. InInternational Conference on Machine Learning (ICML) (Proceedings of Machine Learning Researc...

    2024

  14. [14]

    Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. 2024. DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models. InConference on Empirical Methods in Natural Language Processing (EMNLP). 13487–13521. doi:10.18653/v1/2024.emnlp-main.748

    doi:10.18653/v1/2024.emnlp-main.748 2024

  15. [15]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. FinanceBench: A New Benchmark for Financial Question Answering.arXiv preprint(2023). arXiv:2311.11944 doi:10.48550/arXiv.2311.11944

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.11944 2023

  16. [16]

    Tengjun Jin, Yuxuan Zhu, and Daniel Kang. 2025. ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines.arXiv preprint(2025). arXiv:2504.04808 doi:10.48550/arXiv.2504.04808

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.04808 2025

  17. [17]

    Kimi Team. 2025. Kimi K2: Open Agentic Intelligence.arXiv preprint(2025). arXiv:2507.20534 doi:10.48550/arXiv.2507.20534

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.20534 2025

  18. [18]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettle- moyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. InInterna- tional Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research, Vol. 202). 18319–18345. https://arxi...

    Pith/arXiv arXiv 2023

  19. [19]

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. InInternational Conference on Learning Representations (IC...

    2025

  20. [20]

    Chang, Fei Huang, Reynold Cheng, and Yongbin Li

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.-C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. InAdvances in Neural ...

    Pith/arXiv arXiv 2023

  21. [21]

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu

  22. [22]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. AgentBench: Evaluating LLMs as Agents. InInternational Conference on Learning Represe...

    Pith/arXiv arXiv 2024

  23. [23]

    McKeeman

    William M. McKeeman. 1998. Differential Testing for Software.Digital Technical Journal10, 1 (1998), 100–107

    1998

  24. [24]

    MiniMax. 2026. The MiniMax-M2 Series: Mini Activations Unleashing Max Real- World Intelligence.arXiv preprint(2026). arXiv:2605.26494 doi:10.48550/arXiv. 2605.26494

    Pith/arXiv arXiv doi:10.48550/arxiv.2605.26494 2026

  25. [25]

    MiniMax. 2026. MiniMax M3: Frontier Coding, 1M Context, Native Multimodality. https://www.minimax.io/blog/minimax-m3

    2026

  26. [26]

    OpenAI. 2026. Introducing GPT-5.3-Codex. https://openai.com/index/gpt-5-3- codex/

    2026

  27. [27]

    OpenAI. 2026. Introducing GPT-5.5. https://openai.com/index/introducing-gpt- 5-5/

    2026

  28. [28]

    2025.MySQL: The World’s Most Popular Open Source Database

    Oracle Corporation. 2025.MySQL: The World’s Most Popular Open Source Database. Oracle Corporation. https://www.mysql.com/

    2025

  29. [29]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun

  30. [30]

    Raghav Sethi, Martin Traverso, Dain Sundstrom, David Phillips, Wenlei Xie, Yutian Sun, Nezih Yegitbasi, Haozhun Jin, Eric Hwang, Ning Shang, Sailesh Mikkilineni, and Sameer Paranjpye. 2019. Presto: SQL on Everything. InIEEE International Conference on Data Engineering (ICDE). 1802–1813. doi:10.1109/ ICDE.2019.00196

    arXiv 2019

  31. [31]

    InInternational Conference on Learning Representations (ICLR)

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. InInternational Conference on Learning Representations (ICLR). ICLR 2024 Spotlight. https://arxiv.org/abs/2307.16789

    Pith/arXiv arXiv 2024

  32. [32]

    2025.CodeBuddy: An Agentic Coding and Development Platform

    Tencent. 2025.CodeBuddy: An Agentic Coding and Development Platform. Tencent. https://www.codebuddy.ai/

    2025

  33. [33]

    2025.OpenClaw: Open-Source Personal AI Assistant

    Peter Steinberger. 2025.OpenClaw: Open-Source Personal AI Assistant. https: //github.com/openclaw/openclaw

    2025

  34. [34]

    Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Ning Zhang, Suresh Anthony, Hao Liu, and Raghotham Murthy. 2010. Hive - A Petabyte Scale Data Warehouse Using Hadoop. InIEEE International Conference on Data Engineering (ICDE). 996–1005. doi:10.1109/ICDE.2010.5447738

    arXiv 2010

  35. [35]

    2026.Hy3: A 295B Mixture-of-Experts Large Language Model

    Tencent Hunyuan Team. 2026.Hy3: A 295B Mixture-of-Experts Large Language Model. Tencent. https://github.com/Tencent-Hunyuan/Hy3

    2026

  36. [36]

    Ziting Wang, Shize Zhang, Haitao Yuan, Jinwei Zhu, Wei Dong, and Gao Cong

  37. [37]

    Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2024. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback. InInternational Conference on Learning Representations (ICLR). https://arxiv.org/abs/2309.10691

    Pith/arXiv arXiv 2024

  38. [38]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev

  39. [39]

    Franklin, Scott Shenker, and Ion Stoica

    Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Joseph Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2012. Re- silient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. InUSENIX Conference on Networked Systems Design and Implemen- tation (NSDI). 15–28. https://www.usenix.org/system...

    2012

  40. [40]

    Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, Qi Liu, Zhifang Sui, and Tong Yang

  41. [41]

    arXiv:2604.06132 doi:10.48550/arXiv.2604.06132

    Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents.arXiv preprint(2026). arXiv:2604.06132 doi:10.48550/arXiv.2604.06132

    Pith/arXiv arXiv doi:10.48550/arxiv.2604.06132 2026

  42. [42]

    Low-score

    Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning.arXiv preprint(2017). arXiv:1709.00103 doi:10.48550/arXiv.1709.00103 DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness KDD ’27, August 1–5, 2027, San Jose, CA, USA A Task Listing Ta...

    Pith/arXiv arXiv doi:10.48550/arxiv.1709.00103 2017

  43. [45]

    Qiaohong Zhang, Weihao Ye, Jialong Chen, Yi Luo, Boyuan Li, Bowen Deng, Zibin Zheng, Jianhao Lin, Wei-Shi Zheng, and Chuan Chen. 2025. DataClawBench: An KDD ’27, August 1–5, 2027, San Jose, CA, USA Meng and Yang et al. Agent Benchmark for Exploratory Real-World Financial Data Analysis.arXiv preprint(2025). arXiv:2605.02503 doi:10.48550/arXiv.2605.02503

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.02503 2025

  44. [46]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2306.05685

    Pith/arXiv arXiv 2023

  45. [2018]

    InConference on Empirical Methods in Natural Language Processing (EMNLP)

    Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross- Domain Semantic Parsing and Text-to-SQL Task. InConference on Empirical Methods in Natural Language Processing (EMNLP). 3911–3921. doi:10.18653/v1/ D18-1425

    doi:10.18653/v1/

  46. [2024]

    InInternational Conference on Learning Representations (ICLR)

    Generative Judge for Evaluating Alignment. InInternational Conference on Learning Representations (ICLR). System name: Auto-J. https://arxiv.org/abs/ 2310.05470

    Pith/arXiv arXiv

  47. [2026]

    InACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

    FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data. InACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). KDD ’26, Jeju Island, Republic of Korea. https://arxiv.org/ abs/2509.02473

    Pith/arXiv arXiv