Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DataSpace argues that data agents should be judged by whether they return the exact complete table requested from a heterogeneous workspace, and shows that the best controlled model reaches only 66.34 percent accuracy.

desk verdict A solid, genuinely useful benchmark with a careful construction pipeline; the two things to press on are gold-answer verifiability and missing error bars. read the letter →

arxiv 2608.03451 v1 pith:MI6LS767 submitted 2026-08-04 cs.AI

Boyan Li , Zhuowen Liang , Yupeng Xie , Xiaotian Lin , Tianqi Luo , Xinyu Liu , Yizhang Zhu , Zhangyang Peng
show 6 more authors
Yuan Li Zhengxuan Zhang Jiayi Zhang Nan Tang Guoliang Li Yuyu Luo
This is my paper · ORCID
classification cs.AI
keywords dataagentsbenchmarkheterogeneousworkspacestabularoutputdeterministicevaluationtext-to-SQLmultimodalevidencecross-languageanalytics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DataSpace is a benchmark that asks a data agent to take a single natural-language question and a task-local workspace, find the relevant evidence across CSV, JSON, SQLite, Markdown, PDF, and video, and return one complete typed table. The paper's central claim is that this setting can be evaluated deterministically, without an LLM judge, and that it is not yet saturated: the best controlled model reaches 66.34% Task Accuracy, 76 of 410 tasks are missed by all six tested backbones, and the choice of agent harness alone moves accuracy by 15.36 points. A sympathetic reader should care because the benchmark unifies three properties that earlier suites kept separate: workspace-scale discovery, a complete-table output contract, and reproducible evaluation. The paper also isolates where agents currently fail, pointing to output materialization and cross-modal integration rather than mere evidence retrieval.

What carries the argument

The load-bearing mechanism is DataSpace-Builder, a four-stage construction pipeline that turns executable text-to-SQL instances into heterogeneous workspace tasks. It jointly translates question, database, and SQL across languages, samples a task-local relational instance while preserving primary and foreign keys and query conditions, routes tables to CSV, JSON, SQLite, Markdown, PDF, and query-conditioned video renderers, and then subjects every task to independent expert review and repair. The reference answer is produced by executing the transformed SQL against the sampled database, so the gold tables inherit an execution-grounded origin. The evaluation side uses a deterministic comparator that ignores prediction headers, aligns columns by trying all permutations, canonicalizes types, precision, and units, and compares rows either as ordered sequences or as multisets depending on the task configuration.

What would settle it

Have an independent team reconstruct the reference answer for the 60 released tasks directly from each workspace without seeing the paper's gold tables, then run the official evaluator on both gold sets; if the two gold sets disagree on even a small share of tasks, the reported accuracy numbers are not a stable benchmark property. A second check is to give an agent an oracle that reveals exactly which artifacts are required: if accuracy rises far above 66.34%, the bottleneck is discovery and multimodal reading; if not, the residual failures sit in computation and output materialization.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that workspace-style data agents can be benchmarked by exact tabular output, and that current frontier models are far from solving such tasks. It reports that with a fixed agent harness, the best controlled backbone achieves 66.34% Task Accuracy, the second-best trails by 1.71 points while using 74.2% fewer tokens, and the strongest harness outperforms the weakest by 15.36 points with the same model. It further reports that multimodal tasks underperform single-modal tasks for every backbone by 1.8–14.0 points, that join-requiring tasks drop by 9.7–19.8 points, and that 76 tasks are solved by none of the six backbones while 56 are solved by all. Materialization errors dominate the strongest model's failures: 52.2% of 136 audited failures originate in output materialization, and target-result misunderstanding plus faulty column projection account for 56.6%. The paper's finding is that DataSpace remains unsaturated and that its deterministic evaluator provides a stable target for measuring future progress.

Load-bearing premise

The benchmark's gold answers are only as trustworthy as the source text-to-SQL instances they were built from, plus the translation, rendering, and expert-review steps applied to them, and since only 60 of 410 reference tables are publicly released that chain cannot be fully audited from outside.

Editorial extensions

If this is right

  • If DataSpace's output contract is the right one, partial or factoid answers must be scored wrong even when some returned values are correct.
  • The 15.36-point harness spread with a fixed backbone means benchmark reports must pin down the harness as carefully as the model, or leaderboard comparisons are not meaningful.
  • Because multimodal integration and joins degrade every tested backbone, these operations, not basic retrieval, should be the focus of the next round of data-agent training and tooling.
  • The 56.6% share of failures from output materialization and target-result misunderstanding implies that projects that improve an agent's ability to emit exactly the requested schema may yield larger gains than better source selection.
  • The 76 tasks missed by all six backbones and the 81.46% oracle-union ceiling define a hard core that future benchmark suites can use to measure saturation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to release the remaining 350 reference tables; if independent re-annotation disagrees on a non-negligible share of tasks, the reported accuracy scale would need revisiting.
  • Because video tasks relocate a predicate or answer value into the video, progress in video-language models could saturate those tasks faster than the document tasks, shifting the difficulty mix unevenly over time.
  • The same construction pipeline could generate fresh tasks from new text-to-SQL sources on demand, allowing the benchmark to grow without the expert-review step becoming the limiting cost.
  • The 15.36-point harness spread suggests that stronger agent designs may lift scores substantially before raw model capability becomes the binding constraint, so the 66.34% figure is not a ceiling.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. DataSpace is a new benchmark for data agents that must answer a natural-language question by discovering and combining evidence across task-local heterogeneous workspaces (CSV, JSON, SQLite, Markdown, PDF, and video) and returning a complete tabular result. The paper contributes DataSpace-Builder, a four-stage pipeline that transforms EHRSQL and BULL text-to-SQL instances into 410 cross-language multimodal tasks with reference answers, a deterministic, header-invariant tabular evaluator, and an empirical study of six frontier multimodal backbones and five agent harnesses. The headline findings are that the benchmark remains unsaturated (best controlled accuracy 66.34%, with 76 tasks missed by all backbones), that harness choice causes a 15.36-point spread, and that multimodal integration and joins are the most consistent sources of degradation.

Significance. If the benchmark and its reference answers are sound, DataSpace fills a real gap: it combines heterogeneous evidence discovery, complete tabular output contracts, and deterministic evaluation in a way that most prior benchmarks do not. The construction pipeline is unusually careful: execution-grounded validation, round-trip artifact checks, blind two-reviewer task review, and a deterministic evaluator with explicit type, precision, and order semantics are all credible design choices. The public release of the evaluator, code, and 60 fully gold-annotated tasks is a useful contribution, and the trace-level failure analysis is more informative than typical aggregate error rates. However, the central validity claim rests on the correctness of the reference answers, and that premise is currently not fully auditable: the source SQL is assumed correct, the LLM judge agreement rate is unreported, the human review process is described without inter-reviewer reliability statistics, and 350 of 410 gold tables are withheld. In addition, all headline numbers come from single runs of stochastic API models with no confidence intervals.

major comments (3)
  1. [§4.4, Appendix B] The correctness of every reference answer inherits from the source EHRSQL/BULL gold SQL and from the cross-language transformation, but the validation chain is not auditable from the paper. The execution-equivalence check verifies only that the translated SQL and translated database agree; the only question-level semantic check is the LLM judge in Figure 12, whose agreement, repair, and reject rates are not reported. Section 4.4 and Appendix A.5 describe two-reviewer gold verification, yet no inter-reviewer reliability statistics, repair counts, removal counts, or configuration-disagreement rates are given. Because Task Accuracy in Eq. (9) is exact equivalence to these golds, even a small number of wrong reference answers would bias the 66.34% ceiling and change the membership of the 76-task hard core. Please report the LLM-judge validation statistics and the human-review agreement/repair/removal ledger, or provide an independent audit of a random sample of golds.
  2. [Appendix B] Only 60 of 410 gold tables are released, and the selection criteria for these 60 'representative' tasks are not described. A third party cannot therefore check the withheld 350 golds, including the 76-task hard core that Finding 1 depends on, nor can they reproduce the official 66.34% accuracy from the released artifacts. If the 60 tasks are not a random or stratified sample, local end-to-end evaluation on them gives no valid estimate of full-benchmark behavior. Please release all gold tables and configurations, or justify the withholding with a concrete protocol (e.g., a held-out test set with a fixed public leaderboard) and show that the 60 released tasks are representative along the dimensions used in Section 5.
  3. [§6.1] All headline results are from a single run per backbone–harness pair over stochastic API models. The top two backbones differ by only seven tasks (66.34% vs. 64.63%), and the efficiency comparisons in §6.3 (e.g., 74.2% fewer tokens, 50.3% fewer actions) are computed from the same single runs. Without repeated runs, seeds, temperature-0 determinism checks, or bootstrap confidence intervals, the model ranking and the quantitative efficiency gaps are not statistically grounded. Please report variance across at least 3–5 repeated runs for the main comparisons, or demonstrate that the endpoints are deterministic enough that single runs are adequate.
minor comments (4)
  1. [Figure 1] The illustration contains typos: 'Catetory' should be 'Category' and 'Algin Entities' should be 'Align Entities'.
  2. [§A.7] The experimental setup says all backbone calls use the provider-default reasoning configuration, but does not state temperature, top-p, or any determinism setting. Please specify the sampling parameters or explicitly state that they are left at provider defaults and that this limits reproducibility.
  3. [Appendix B] Please state how the 60 released tasks were selected (e.g., random stratified by domain and modality) and whether the withheld 350 are fixed and versioned, so that external evaluations can be compared across time.
  4. [§4.3] For the video predicate-abstraction strategy, the paper says the question is rejected when a selected span cannot be removed without changing semantics, but no statistics are reported on how often this occurs or how many video-augmented tasks required repair. A sentence with these counts would help calibrate the reliability of the video evidence chain.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: gold answers are execution-derived and blind-reviewed; the headline accuracy is an external measurement, not a fitted or self-cited result.

full rationale

The paper's load-bearing claim (Finding 1, that DataSpace remains unsaturated) is an empirical measurement over a frozen benchmark, not a quantity derived by fitting the construction pipeline to the evaluated models. The derivation chain is: source Text-to-SQL instances (D0,q0,sigma0) are transformed via cross-language transformation into (Dc,qc,sigmac); constraint-aware sampling produces Ds and the candidate gold Ys = Exec(Ds,sigmac), per Eq. 5; modality routing and rendering are validated by round-trip, coverage, and cell-fidelity checks; two blind expert reviewers must independently agree on the gold and the evaluation configuration before a task is released. The evaluation then compares an agent's submitted table to the frozen reference using the deterministic equivalence in Eq. 8 and reports Task Accuracy in Eq. 9. No step in this chain uses any agent output or model performance as an input, and no parameter is fitted to the six backbones or five harnesses reported in Table 3. The construction-time LLM judge and human review are quality gates that can fail, which is a data-quality and external-verifiability risk (only 60 of 410 golds are released, and no inter-reviewer reliability statistics are reported), but they are not circular in the derivation sense. Self-citations such as 'Following DataMagic [50]' borrow a rendering method, but they are not load-bearing for the unsaturation conclusion, which rests on the controlled empirical comparison. Thus no circular step is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the correctness of source SQL, on probabilistic LLM translation and validation, and on human review. None are fitted to agent performance, so circularity is low, but the benchmark's validity depends on these audited assumptions.

free parameters (6)
  • theta_ov (value-overlap merge threshold)
    Used in translation-unit extraction (Equation 10, A.1.2) to decide when columns merge into a translation cluster; chosen by configuration, not estimated from data.
  • k_min (minimum shared values for cluster merge)
    Second condition for cluster merging in A.1.2; prevents a single common token from connecting unrelated columns.
  • Per-table soft row budgets b_R
    Part of the sampling policy in Section 4.2; controls how many rows each table contributes to the task-local workspace and affects answer content, but is not fit to model performance.
  • Sampling attempt budget K = 3
    Maximum resampling attempts per task in A.2.1; hand-set construction parameter.
  • Seeded randomness z and z_r
    Seeds for relational sampling and base routing (A.2.1, A.3.1); not fit to data, but they influence which rows and renderers appear.
  • Per-column evaluation configuration (type, precision, percent convention, order flag)
    Authored by human reviewers (Section 4.4, A.6) and defines what counts as a correct answer; if misconfigured, accuracy scores change. The paper reports no inter-reviewer agreement statistics.
assumptions (5)
  • domain assumption The source EHRSQL and BULL questions and SQL are correct and encode the intended analytical semantics.
    DataSpace-Builder transforms these tuples and derives gold answers by executing the SQL on sampled databases (Section 4.1); any source-SQL error propagates to the benchmark gold.
  • domain assumption LLM translation and LLM-based question-SQL alignment preserve meaning across languages.
    Cross-Language Transformation validates execution equivalence and uses an LLM judge for semantic alignment (Section 4.1); this is a probabilistic check, not a proof.
  • domain assumption Human expert review guarantees task unambiguity and gold correctness.
    Two blind reviewers per task, consensus repair, and rechecking (Section 4.4, A.5) are the final authority; no inter-reviewer reliability statistics are reported.
  • domain assumption Generated documents and videos faithfully carry the required evidence without leakage or ambiguity.
    Grounded generation, cell-to-span validation, and human review are relied on; rendering errors in documents or videos would contaminate the gold.
  • domain assumption No external knowledge is required to answer tasks; the workspace is self-contained.
    The task formulation gives agents only the question and workspace (Equation 1), so real-world entities and alert rules must be identifiable from workspace artifacts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces." pith.science (2026). https://pith.science/paper/MI6LS767

@misc{pith2026260803451,
  author       = {Pith},
  title        = {Pith review of: DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MI6LS767}},
  note         = {Machine review of arXiv:2608.03451}
}
read the original abstract

Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video. DataSpace also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We construct DataSpace with DataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show that DataSpace remains unsaturated and identify key challenges for improving data-agent reliability.

Figures

Figures reproduced from arXiv: 2608.03451 by the authors.

Figure 1
Figure 1. The DataSpace task interface, illustrated with a fund-risk task. The agent combines an alert rule from video, category benchmarks extracted from a long PDF, and daily NAV queried from SQLite, then aligns entities, computes the requested metrics, and returns the complete tabular result. question rarely resides in a single clean table or a preselected data￾base [7, 35, 48]. The question and its evidence may cross lang… view at source ↗
Figure 2
Figure 2. Overview of DataSpace-Builder. Text-to-SQL instances pass through Cross-Language Transformation, Constraint￾Aware Relational Sampling, Modality Routing & Artifact Rendering, and Human Review & Task Repair before being frozen as heterogeneous benchmark records. predicate values, and executable SQL. We therefore treat cross￾language transformation (CLT) as a joint migration of the ques￾tion, database state, and execut… view at source ↗
Figure 3
Figure 3. Workspace availability and annotated solution evi [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Analytical-operation coverage and compositional [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Backbone accuracy–efficiency trade-offs with [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Error analysis of 136 Grok 4.5 failures. Bars show [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Prompt for schema mapping. SYSTEM Translate database values from <SOURCE_LANGUAGE> to <TARGET_LANGUAGE> while preserving equality and join semantics. RULES 1. One source value has exactly one target value within this cluster. 2. Use the same target for every listed tab…
Figure 10
Figure 10. Figure 10: Prompt for clustered-value mapping. Updates are scoped by the source table and column recorded in 𝑀; global string replacement is never applied to database contents. When a source value is a substring of another value, exact matching is used for cells and longest-firs…
Figure 11
Figure 11. Figure 11: Prompt for question translation. SYSTEM Audit whether a translated analytical question preserves the source intent and remains answered by the translated SQL. Do not solve the query. CHECK 1. Requested output entities and fields. 2. All filters, values, comparison dir…
Figure 12
Figure 12. Figure 12: Prompt for question–SQL alignment. A.1.7 Validation Gates and Failure Handling. Validation proceeds from inexpensive structural checks to semantic checks: (1) Map validation checks coverage, scope, protected identities, and table/column target-name collisions. (2) Dat…
Figure 13
Figure 13. Figure 13: Prompt for fact-grounded document generation. [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Source-to-document alignment for one sampled [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Prompt for evidence-grounded storyboard plan [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Representative frames from the released task videos. Task 193 distributes its security scope and predicate specification [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Illustrative end-to-end reviewer trace in the human-review system. Panel (a) shows the blind phase, in which the [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Frozen evaluation configuration for Task 120. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Doc2DB-Bench provides 203 long-document, database pairs to test whether LLMs can reconstruct multi-table relational databases with correct keys, relationships, and constraints.

Reference graph

Works this paper leans on

94 extracted references · 53 canonical work pages · cited by 1 Pith paper

  1. [1]

    MiMo-V2.5

    2026. MiMo-V2.5. https://huggingface.co/collections/XiaomiMiMo/mimo-v25

    2026

  2. [2]

    Anthropic. 2026. How Claude Code Works. Accessed July 27, 2026. https: //code.claude.com/docs/en/how-claude-code-works

    2026

  3. [3]

    Anthropic. 2026. Introducing Claude Sonnet 5. https://www.anthropic.com/ news/claude-sonnet-5

    2026

  4. [4]

    Yutong Bian, Xianhao Lin, Yupeng Xie, Tianyang Liu, Mingchen Zhuge, Siyuan Lu, Haoming Tang, Jinlin Wang, Jiayi Zhang, Jiaqi Chen, et al. 2025. You Don’t Know Until You Click: Automated GUI Testing for Production-Ready Software Evaluation.arXiv preprint arXiv:2508.14104(2025)

    arXiv 2025

  5. [5]

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. HybridQA: A Dataset of Multi-Hop Question An- swering over Tabular and Textual Data. InEMNLP (Findings) (Findings of ACL, Vol. EMNLP 2020). Association for Computational Linguistics, 1026–1036

    2020

  6. [6]

    Yiyu Chen, Yifan Wu, Shuyu Shen, Yupeng Xie, Leixian Shen, Hui Xiong, and Yuyu Luo. 2025. ChartMark: A Structured Grammar for Chart Annotation. In 2025 IEEE Visualization and Visual Analytics (VIS). IEEE, 311–315

    2025

  7. [7]

    Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf. 2025. Dabstep: Data agent benchmark for multi-step reasoning.arXiv preprint arXiv:2506.23719(2025)

    arXiv 2025

  8. [8]

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. 2025. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysi...

    2025

Show all 94 references
  1. [9]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. FinanceBench: A New Benchmark for Financial Question Answering.CoRRabs/2311.11944 (2023)

    2023 arXiv

  2. [10]

    KDD Cup 2026. 2026. KDD Cup 2026: Data Agents for Complex Data Analysis. Accessed July 27, 2026. https://dataagent.top/

    2026

  3. [11]

    Zabreyko, Chenning Li, Ferdi Kossmann, Jialin Ding, Jun Chen, Markos Markakis, Matthew Russo, Weiyang Wang, Ziniu Wu, Michael J

    Eugenie Lai, Gerardo Vitagliano, Ziyu Zhang, Sivaprasad Sudhir, Om Chabra, Anna Zeng, Anton A. Zabreyko, Chenning Li, Ferdi Kossmann, Jialin Ding, Jun Chen, Markos Markakis, Matthew Russo, Weiyang Wang, Ziniu Wu, Michael J. Cafarella, Lei Cao, Samuel Madden, and Tim Kraska. 20...

    2025

  4. [12]

    Gyubok Lee, Hyeonji Hwang, Seongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang, Minjoon Seo, Jong-Yeup Kim, and Edward Choi. 2023. EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records.CoRR abs/2301.07695 (2023)

    2023

  5. [13]

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating Language Models on Real-World Enterprise T...

    2025

  6. [14]

    Boyan Li, Chong Chen, Zhujun Xue, Yinan Mei, and Yuyu Luo. 2025. DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Framework.CoRR abs/2510.17586 (2025)

    2025 arXiv

  7. [15]

    Boyan Li, Ou Ocean Kun Hei, Yue Yu, and Yuyu Luo. 2026. DPC: Training-Free Text-to-SQL Candidate Selection via Dual-Paradigm Consistency. InACL (1). Association for Computational Linguistics, 6897–6913

    2026

  8. [16]

    Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024. The Dawn of Natural Language to SQL: Are We Fully Ready? [Experiment, Analysis & Benchmark ].Proc. VLDB Endow.17, 11 (2024), 3318–3331

    2024

  9. [17]

    Boyan Li, Yiran Peng, Yupeng Xie, Sirong Lu, Yizhang Zhu, Xing Mu, Xinyu Liu, and Yuyu Luo. 2026. DeepEye: A Steerable Self-driving Data Agent System. InCompanion of the International Conference on Management of Data(India) (SIGMOD Companion ’26). Association for Computing Mac...

    2026

  10. [18]

    Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo

  11. [19]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chen-Chuan Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A BIg Be...

    2023

  12. [20]

    Yiyang Li, Zheyuan Zhang, Tianyi Ma, Zehong Wang, Keerthiram Murugesan, Chuxu Zhang, and Yanfang Ye. 2026. LongDA: Benchmarking LLM Agents for Long-Document Data Analysis.arXiv preprint arXiv:2601.02598(2026)

    2026

  13. [21]

    Zhuowen Liang, Xiaotian Lin, Zhengxuan Zhang, Yuyu Luo, Haixun Wang, and Nan Tang. 2026. Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs.CoRRabs/2603.29232 (2026)

    2026

  14. [22]

    Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2025. A Survey of Text-to-SQL in the Era of LLMs: Where Are We, and Where Are We Going?IEEE Trans. Knowl. Data Eng.37, 10 (2025), 5735–5754

    2025

  15. [23]

    Xinyu Liu, Shuyu Shen, Boyan Li, Nan Tang, and Yuyu Luo. 2025. NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation. InKDD (2). ACM, 5662–5673

    2025

  16. [24]

    Tianqi Luo, Chuhan Huang, Leixian Shen, Boyan Li, Shuyu Shen, Wei Zeng, Nan Tang, and Yuyu Luo. 2025. nvBench 2.0: Resolving Ambiguity in Text-to- Visualization through Stepwise Reasoning. InNeurIPS

    2025

  17. [25]

    Peixian Ma, Boyan Li, Runzhi Jiang, Ju Fan, Nan Tang, and Yuyu Luo. 2024. A Plug-and-Play Natural Language Rewriter for Natural Language to SQL.CoRR abs/2412.17068 (2024)

    2024 arXiv

  18. [26]

    Ruiying Ma, Shreya Shankar, Ruiqi Chen, Yiming Lin, Sepanta Zeighami, Ra- joshi Ghosh, Abhinav Gupta, Anushrut Gupta, Tanmai Gopal, and Aditya G Parameswaran. 2026. Can ai agents answer your data questions? a benchmark for data agents.arXiv preprint arXiv:2603.20576(2026)

    2026

  19. [27]

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. 2024. MMLONGBENCH-DOC: Bench- marking Long-context Document Understanding with Visual...

    2024

  20. [28]

    MiniMax. 2026. MiniMax M3: Frontier Coding, 1M Context, Native Multimodality—All in One Model. https://www.minimax.io/blog/minimax-m3

    2026

  21. [29]

    Moonshot AI. 2026. Kimi K3: Open Frontier Intelligence. https://www.kimi. com/blog/kimi-k3

    2026

  22. [30]

    Arik, and Tomas Pfister

    Jaehyun Nam, Jinsung Yoon, Jiefeng Chen, Jinwoo Shin, Sercan Ö. Arik, and Tomas Pfister. 2025. MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement.CoRRabs/2506.15692 (2025)

    2025 arXiv

  23. [31]

    OpenAI. 2026. Codex CLI. Accessed July 27, 2026. https://learn.chatgpt.com/ docs/codex/cli

    2026

  24. [32]

    OpenAI. 2026. GPT-5.6 Sol. Accessed July 27, 2026. https://developers.openai. com/api/docs/models/gpt-5.6-sol

    2026

  25. [33]

    Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structured Tables. InACL (1). The Association for Computer Linguistics, 1470–1480

    2015

  26. [34]

    Wenqi Pei, Shizheng Hou, Boyan Li, Chen Han, Zhichao Shi, and Yuyu Luo. 2026. ROSE: An Intent-Centered Evaluation Metric for NL2SQL. InACL (1). Association for Computational Linguistics, 5682–5709

    2026

  27. [35]

    Ruyi Qi, Zhou Liu, and Wentao Zhang. 2026. DataCross: A Unified Benchmark and Agent Framework for Cross-Modal Heterogeneous Data Analysis.CoRR abs/2601.21403 (2026)

    2026

  28. [36]

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. 2025. ‘smolagents‘: a smol library to build great agentic systems. https://github.com/huggingface/smolagents

    2025

  29. [37]

    Zhihao Shuai, Boyan Li, Siyu Yan, Yuyu Luo, and Weikai Yang. 2026. DeepVIS: Bridging Natural Language and Data Visualization Through Step-Wise Reason- ing.IEEE Trans. Vis. Comput. Graph.32, 1 (2026), 868–878

    2026

  30. [38]

    SpaceXAI. 2026. Grok Build. Accessed July 27, 2026. https://docs.x.ai/build/ overview

    2026

  31. [39]

    SpaceXAI. 2026. Introducing Grok 4.5. https://x.ai/news/grok-4-5

    2026

  32. [40]

    Ji Sun, Guoliang Li, Peiyao Zhou, Yihui Ma, Jingzhe Xu, and Yuan Li. 2025. AgenticData: An Agentic Data Analytics System for Heterogeneous Data.CoRR abs/2508.05002 (2025)

    2025 arXiv

  33. [41]

    Zhaoyan Sun, Shan Zhong, Daizhou Wen, Jiaxing Han, Guoliang Li, Ying Yan, Peng Zhang, Yu Su, Xiang Qi, Baolin Sun, et al . 2026. AgenticDataBench: A Comprehensive Benchmark for Data Agents.arXiv preprint arXiv:2607.01647 (2026)

    2026 arXiv

  34. [42]

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. MultiModalQA: complex question answering over text, tables and images. InICLR. OpenRe- view.net

    2021

  35. [43]

    Yinghao Tang, Xueding Liu, Boyuan Zhang, Tingfeng Lan, Yupeng Xie, Jiale Lao, Yiyao Wang, Haoxuan Li, Tingting Gao, Bo Pan, et al . 2026. IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation.arXiv preprint arXiv:2601.04498(2026)

    2026 arXiv

  36. [44]

    Yinghao Tang, Yupeng Xie, Yingchaojie Feng, Tingfeng Lan, Jiale Lao, Yue Cheng, and Wei Chen. 2026. ViviDoc: Generating Interactive Documents through Human-Agent Collaboration.arXiv preprint arXiv:2603.27991(2026)

    2026

  37. [45]

    Teable. 2026. Teable AI: Overview. Accessed July 27, 2026. https://help.teable.ai/ en/basic/ai/overview

    2026

  38. [46]

    Vercel. 2026. AI Gateway. Accessed July 27, 2026. https://vercel.com/docs/ai- gateway

    2026

  39. [47]

    Jiayi Wang and Guoliang Li. 2025. AOP: Automated and Interactive LLM Pipeline Orchestration for Answering Complex Queries. InCIDR. www.cidrdb.org

    2025

  40. [48]

    Ziting Wang, Shize Zhang, Haitao Yuan, Jinwei Zhu, Shifu Li, Wei Dong, and Gao Cong. 2025. FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data.CoRRabs/2509.02473 (2025)

    2025 arXiv

  41. [49]

    Yupeng Xie, Yuyu Luo, Guoliang Li, and Nan Tang. 2024. Haichart: Human and AI paired visualization system.arXiv preprint arXiv:2406.11033(2024)

    2024 arXiv

  42. [50]

    Yupeng Xie, Chen Ma, Zhenyang Wang, Liangwei Wang, Jiayi Zhu, Chuxuan Zeng, Zhouan Shen, Boyan Li, and Yuyu Luo. 2026. DataMagic: Transforming Conference’17, July 2017, Washington, DC, USA Li et al. Tabular Data into Data Insight Video.CoRRabs/2606.20388 (2026)

    2026 arXiv

  43. [51]

    Yupeng Xie, Zhiyang Zhang, Yifan Wu, Sirong Lu, Jiayi Zhang, Zhaoyang Yu, Jinlin Wang, Sirui Hong, Bang Liu, Chenglin Wu, et al. 2025. Visjudge-bench: Aes- thetics and quality assessment of visualizations.arXiv preprint arXiv:2510.22373 (2025)

    2025

  44. [52]

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue...

    2024 arXiv

  45. [53]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. InEMNLP. Association for Computational Linguistics, 2369–2380

    2018

  46. [54]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InICLR. OpenReview.net

    2023

  47. [55]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. I...

    2018

  48. [56]

    Chao Zhang, Yuren Mao, Yijiang Fan, Yu Mi, Yunjun Gao, Lu Chen, Dongfang Lou, and Jinshu Lin. 2024. FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis. InSIGMOD Conference Companion. ACM, 93–105

    2024

  49. [57]

    Chao Zhang, Shaolei Zhang, Quehuan Liu, Sibei Chen, Tong Li, and Ju Fan

  50. [58]

    Shaolei Zhang, Ju Fan, Meihao Fan, Guoliang Li, and Xiaoyong Du. 2025. DeepAnalyze: Agentic Large Language Models for Autonomous Data Science. arXiv:2510.16872 [cs.AI] https://arxiv.org/abs/2510.16872

    2025

  51. [59]

    TAIJI: MCP-based Multi-Modal Data Analytics on Data Lakes.CoRR abs/2505.11270 (2025)

    2025 arXiv

  52. [60]

    Yizhang Zhu, Runzhi Jiang, Boyan Li, Nan Tang, and Yuyu Luo. 2025. El- lieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing.CoRR abs/2503.22402 (2025)

    2025 arXiv

  53. [61]

    Zhengxuan Zhang, Zhuowen Liang, Yin Wu, Teng Lin, Yuyu Luo, and Nan Tang. 2025. DataMosaic: Explainable and Verifiable Multi-Modal Data Analytics through Extract-Reason-Verify.CoRRabs/2504.10036 (2025)

    2025

  54. [62]

    tables": [ {

    Yizhang Zhu, Liangwei Wang, Chenyu Yang, Xiaotian Lin, Boyan Li, Wei Zhou, Xinyu Liu, Zhangyang Peng, Tianqi Luo, Yu Li, Chengliang Chai, Chong Chen, Shimin Di, Ju Fan, Ji Sun, Nan Tang, Fugee Tsung, Jiannan Wang, Chenglin Wu, Yanwei Xu, Shaolei Zhang, Yong Zhang, Xuanhe Zhou,...

    2025

  55. [63]

    Yizhang Zhu, Zhangyang Peng, Boyan Li, and Yuyu Luo. 2026. SQLConduc- tor: Search-to-Policy Learning for Step-wise Text-to-SQL Orchestration.CoRR abs/2606.23537 (2026)

    2026 arXiv

  56. [65]

    Translate only human-readable table and column names

  57. [66]

    Preserve IDs, standardized codes, SQL keywords, and protected terms

  58. [67]

    Preserve meaning, domain terminology, granularity, and abbreviations

  59. [68]

    Column targets must be unique within their table

    Table targets must be unique in the database. Column targets must be unique within their table

  60. [69]

    items": [ {

    Return JSON only. Do not add, remove, merge, or split input items. INPUT DOMAIN: <DATABASE_DOMAIN> PROTECTED_GLOSSARY: <PROTECTED_GLOSSARY> SCHEMA_ITEMS: <LIST_OF_ITEMS_WITH_STABLE_IDS_TABLE_CONTEXT_AND_DESCRIPTIONS> OUTPUT SCHEMA { "items": [ {"id": "<INPUT_ID>", "target": "<...

  61. [70]

    One source value has exactly one target value within this cluster

  62. [71]

    Use the same target for every listed table/column occurrence

  63. [72]

    Do not translate identifiers, codes, URLs, dates, numbers, or entries marked PROTECTED

  64. [73]

    Preserve units, signs, precision, entity identity, and domain meaning

  65. [74]

    cluster_id

    Return JSON only and include every input ID exactly once. CONTEXT DOMAIN: <DATABASE_DOMAIN> CLUSTER_ID: <CLUSTER_ID> COLUMNS: <TABLE_COLUMN_OCCURRENCES> SCHEMA_GLOSSARY: <RELEVANT_SCHEMA_MAP> VALUES: <VALUES_WITH_STABLE_IDS_AND_PROTECTION_FLAGS> OUTPUT SCHEMA { "cluster_id": "...

    2017

  66. [75]

    Use the supplied target-language glossary for schema and value terms

  67. [76]

    Write a natural user question; do not mention SQL, schemas, or this task

  68. [77]

    Do not add explanations, assumptions, or answer values

  69. [78]

    question

    Return JSON only. INPUT SOURCE_QUESTION: <QUESTION> FOCUSED_GLOSSARY: <RELEVANT_TABLE_COLUMN_VALUE_MAP> OPTIONAL_SQL_ALIGNMENT_CONTEXT: <GOLD_SQL_OR_OPERATOR_SIGNATURE> OUTPUT SCHEMA { "question": "<TRANSLATED_QUESTION>", "used_mapping_ids": ["<ID>"] } Figure 11: Prompt for qu...

  70. [79]

    Requested output entities and fields

  71. [80]

    All filters, values, comparison directions, and negations

  72. [81]

    Aggregation, grouping, DISTINCT semantics, ordering, ranking, and limits

  73. [82]

    Time windows, boundary inclusivity, units, and numerical scale

  74. [83]

    status":

    Consistency with the supplied schema/value glossary. INPUT SOURCE_QUESTION: <SOURCE_QUESTION> TRANSLATED_QUESTION: <TRANSLATED_QUESTION> SOURCE_SQL: <SOURCE_SQL> TRANSLATED_SQL: <TRANSLATED_SQL> FOCUSED_GLOSSARY: <RELEVANT_MAP> OUTPUT SCHEMA { "status": "pass | repair | reject...

    2017

  75. [84]

    Express every REQUIRED_CELL_ID with its exact entity association

  76. [85]

    Do not merge values from different rows

    Preserve identifiers, categorical values, numeric precision, dates, times, signs, and units. Do not merge values from different rows

  77. [86]

    Never infer a missing value

    Follow NULL_POLICY. Never infer a missing value

  78. [87]

    You may add connective prose or domain-neutral scene detail, but it must not create a competing value for any protected field

  79. [88]

    Keep ANCHOR_FIELDS explicit so this section can be joined with other sections about the same records

  80. [89]

    section":

    Return only the requested section and the cell-to-span alignment JSON. INPUT STYLE: <STYLE_SPECIFICATION> ANCHOR_FIELDS: <ANCHOR_COLUMNS_AND_VALUES> ATTRIBUTE_CLUSTER: <COLUMN_NAMES_TYPES_AND_DESCRIPTIONS> ROWS: <TYPED_VALUES_WITH_CELL_IDS> REQUIRED_CELL_IDS: <CELL_IDS> NULL_P...

    2017

  81. [90]

    Every EVIDENCE_ID must appear in at least one scene and must retain its typed value, operator, precision, unit, and entity association

  82. [91]

    Answer evidence must remain readable but need not be adjacent

    Predicate abstraction must communicate both comparison direction and threshold. Answer evidence must remain readable but need not be adjacent

  83. [92]

    Do not invent an alternative rule or a second valid answer

    Use only supplied records for data-bearing distractors. Do not invent an alternative rule or a second valid answer

  84. [93]

    Narration should support the business scenario; do not read out all evidence when doing so would make visual structure unnecessary

  85. [94]

    scenes": [{

    Return JSON only. Every visible constant must identify its source atom. INPUT TASK_STRATEGY: <PREDICATE_ABSTRACTION_OR_ANSWER_EVIDENCE> EVIDENCE_ATOMS: <TYPED_ATOMS_WITH_STABLE_IDS> SUPPORTING_RECORDS: <BOUNDARY_AND_DISTRACTOR_RECORDS> VISUAL_COMPONENT_LIBRARY: <AVAILABLE_COMP...

    2017

  86. [2025]

    InICML (Proceedings of Machine Learning Research, Vol

    Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search. InICML (Proceedings of Machine Learning Research, Vol. 267). PMLR / OpenReview.net

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.