Pith. sign in

REVIEW 3 major objections 5 minor 45 references

Current LLMs handle simple table tasks well but fail on complex, noisy multi-turn data work.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:42 UTC pith:MMOWAPYD

load-bearing objection A genuinely useful benchmark for interactive table-processing LLMs, but the online simulator's credibility rests on a 10-person study and unstated parameters—treat the headline degradation result as provisional until they are reported and probed. the 3 major comments →

arxiv 2608.00018 v1 pith:MMOWAPYD submitted 2026-06-29 cs.DB cs.AI

CITBench: A Comprehensive Benchmark for Interactive Tabular Data Processing with LLMs

Zihan Nan , Yang Gu , Wei Liu , Xi Yan , Zhou Liu , Hao Liang , Wentao Zhang This is my paper
classification cs.DB cs.AI
keywords LLM benchmarktabular data processinginteractive evaluationmulti-turn interactioncognitive simulatordata cleaningtable transformationstate tracking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CITBench is a benchmark with 1,296 instances across 18 tabular task types and four categories—matching, cleaning, augmentation, and transformation—built to test LLMs in both offline and online settings. The paper's central claim is that while current models perform well on simple tables and fully specified rules, they degrade significantly as table complexity rises, rule dependencies tighten, and simulated multi-turn user interactions become noisy. This is the first systematic evaluation of online and multi-turn interactions for tabular data processing agents. Evaluating 13 open-source and closed-source models, the authors find a consistent pattern of decline—most sharply on multi-step and noisy interactive tasks—indicating that LLMs lack the state tracking, planning consistency, and noise resilience needed for realistic data work.

Core claim

The paper claims that LLM-based tabular data assistants are far less capable in realistic interactive use than in single-turn, fully specified tasks. CITBench decomposes tabular work into four high-level categories and generates tasks with verifiable ground truths. Its online mode uses a Cognitive Simulator, calibrated from a 10-participant user study, to model bounded human memory and conversational noise—fuzzy instructions, biased rules, redundancy, and order swaps—that are corrected in subsequent turns. Across 13 models, the authors find a consistent and often steep accuracy decline as complexity and noise increase, with the sharpest drops on multi-step and noisy interactive settings. The

What carries the argument

The Cognitive Simulator with two modules: the Cognitive-Load Window (limits active subtasks per turn to a capacity K) and the Perturbation-Resolution Cycle (injects noise with decaying probability p_t = p0 * exp(-t/tau) and automatically queues corrective subtasks for the next turn). This simulates progressive elicitation, user mistakes, and immediate corrections. The offline pipeline's synthesis-and-verification engine—using cross-model consensus filtering and a two-stage human validation—provides the ground-truth base that both offline and online settings build on.

Load-bearing premise

The online evaluation's realism rests on the Cognitive Simulator's noise model, which is calibrated from a controlled user study of only 10 participants; if real users' conversational behavior in actual deployment differs significantly from the simulated fuzzy, biased, redundant, and order-swapped instructions, the reported performance drops under cognitive noise may not carry over.

What would settle it

Record real interaction logs from a larger and more diverse group of users performing the same 1,296 tasks in a conversational data tool, compare the noise-type frequencies, turn lengths, and correction patterns against simulator outputs, and rerun the same models on those real logs; if the degradation pattern or ranking shifts substantially, the simulator is not a faithful proxy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Benchmarking LLM data assistants must move beyond single-turn, fully specified instructions; interactive, multi-turn evaluation becomes a standard requirement.
  • Accuracy on online tasks trades off with token cost: context reconciliation swells computational footprints as conversational turns accumulate.
  • Current models are fragile under conversational noise, suggesting that noise-aware training—exposing models to contradictory or redundant instructions—could yield outsized gains.
  • The performance inversion between offline and online rankings indicates that interactive capability (intent tracking, error recovery) is partially distinct from static code generation skill.
  • Agent frameworks built on top of base LLMs do not overcome the backbone's table-structure perception limits in tightly coupled workflows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A larger, more diverse user study could shift the noise model; the reported degradation magnitudes might not transfer to other user populations or interaction styles.
  • The offline tasks are generated by a single LLM and filtered by cross-model consensus; regenerating tasks with a different generator and checking whether model rankings and difficulty tiers remain stable would test the benchmark's generalizability.
  • A falsifiable extension would be to compare the simulated online trajectories with logged human–agent interactions on the same underlying tasks; systematic mismatches in turn structure or noise frequencies would require reinterpreting the online results.
  • The observation that small models collapse on multi-step tasks while large models degrade more gracefully suggests an emergent planning-capacity threshold; specialized fine-tuning on interactive data workflows may shift this threshold more than further scaling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. CITBench is a benchmark for evaluating LLMs on interactive tabular data processing. It contributes a taxonomy of 18 task types across four categories (matching, cleaning, augmentation, transformation) and 1,296 instances, with both an offline (single-turn, fully specified) and an online (multi-turn, noisy interactive) evaluation protocol. The offline pipeline uses LLM-based synthesis with cross-model consensus filtering and human validation; the online protocol uses a Cognitive Simulator, calibrated from a 10-participant user study, that injects fuzzy, biased, redundant, and order-swapped instruction perturbations into multi-turn trajectories. The authors evaluate 13 open- and closed-source models, reporting that performance degrades with rule complexity, multi-input/multi-step structure, and simulated conversational noise, and they analyze failure modes and token footprint. The central abstract claim is that current models perform well on simple tables and rules but degrade significantly with table complexity, tighter rule dependencies, and noisy multi-turn interactions.

Significance. If the results hold, CITBench is a valuable addition to the benchmark landscape: it is the first benchmark that systematically targets online, multi-turn interactions for tabular data processing, and it combines a broad task taxonomy with executable ground-truth verification. The paper ships a public code/data release, includes human validation of ground truth, and evaluates a wide suite of modern LLMs. The offline findings — that multi-step pipelines and cross-table dependencies cause large drops in exact-completion rates — are useful and largely consistent with prior work. The paper also makes a useful distinction between conversational multi-turn decomposition (which can help) and noisy perturbations (which hurt). However, the external validity of the headline online results depends almost entirely on the Cognitive Simulator, whose parameters and calibration are not fully reported, and the empirical comparisons lack error bars. These issues need to be addressed before the benchmark's diagnostic claims can be accepted at face value.

major comments (3)
  1. [§IV-C and Algorithm 1] The online evaluation's central claim — that noisy multi-turn interactions degrade performance — rests entirely on the Cognitive Simulator. Yet the simulator's key parameters (p0, τ, K) are never reported, and no sensitivity analysis is provided. The perturbation-resolution cycle (Algorithm 1, lines 9–11) always appends a corrective subtask to the head of the next turn's queue, so every perturbation is immediately followed by a repair. This imposes a very specific interaction pattern that may not generalize to real users, who may not notice errors or may correct them in varied ways. The calibration is based on a controlled user study of only 10 participants (Sec. III-C), which is too small to reliably estimate the four-profile categorical distribution or the decay p_t = p0 exp(-t/τ). The paper must report fitted parameter values, uncertainty, and must show that the qualitative result in
  2. [§V-A-c and Table III / Fig. 6] The evaluation uses only a 50% balanced slice of offline tasks and a 25% uniform slice of online sequences, but the paper reports single point estimates without confidence intervals, error bars, or significance tests. The metric definition uses K independent inference trials (ar P = 1/K Σ S_i) but the value of K is never stated. Consequently, headline differences such as 'GLM-4.7 outperforms Gemini-3-Pro by an absolute margin of 7%' (Fig. 6) and the online-vs-offline inversions may be within sampling noise. The paper should report variance across seeds/bootstrap samples, specify K, and provide a statistical test (or at least error bars) for the key cross-model and online-vs-concat comparisons.
  3. [§IV-B-a and §IV-B-c] The ground-truth construction pipeline uses LLM-generated candidate tasks and reference code, filtered by consensus among six LLMs (Claude-Opus-4.5, Claude-Sonnet-4.5, Gemini-3-Pro, GPT-5.2, GLM-4.7, DeepSeek-V3.2) — several of which are later evaluated as baselines. The paper asserts that rejected tasks stem 'entirely from minor instruction-level layout or description ambiguities' and that this 'confirms that our consensus protocol isolates objective description quality rather than introducing model-contingent filtering artifacts,' but no quantitative evidence is provided (e.g., counts of rejected tasks, category breakdown, examples). Similarly, the human validation protocol is described as a 'double-pass' by 3 experts, but no inter-validator agreement or number of excluded instances is reported. These missing details matter because the consensus filter can shape the difficulty distribu
minor comments (5)
  1. [Table I] Typo: 'SpreedsheetBench' should be 'SpreadsheetBench'. Also, the table's use of '!' marks seems inconsistent with the legend; please clarify.
  2. [§IV-C, Algorithm 1] The resolution types T_c, T_m, T_re are used in Algorithm 1 but defined only informally in the caption of Fig. 4. Please define them explicitly in the text.
  3. [§V-A-c] The '50% balanced slice' is not described: balanced with respect to what (taxonomy, difficulty, domain)? Clarify the sampling procedure and whether online sequences are sampled uniformly over the full distribution.
  4. [§V-B-b / Fig. 9] The text says 'almost all models exhibit significant performance degradations' in the online setting, but Fig. 9 shows Gemini-3-Pro with a 'minor 3.0% counter-trend increase' in ar C. Please temper 'universal' claims and discuss this exception.
  5. [§III-C] The user study is described only in a short paragraph. Please provide details: how tasks were selected, what 'state-of-the-art conversational tabular AI systems' were used, how the noise profiles were annotated, and the inter-annotator agreement.

Circularity Check

0 steps flagged

No significant circularity: the benchmark's correctness criteria are externally anchored by human verification, and the headline degradation is an empirical measurement rather than a construction artifact.

full rationale

CITBench's central claims are benchmark results, not derived quantities. Offline ground truth is generated by a synthesis engine and then subjected to cross-model consensus plus a two-stage expert human validation protocol ('Validators perform exhaustive, cell-by-cell matrix comparisons against the formalized requirements'), so the targets are not defined by the evaluated models' outputs alone. The reported degradation with table complexity and rule dependencies is measured by executing model-generated code in a sandbox against locked ground-truth tables; no equation in the paper forces a particular model to score lower on Complex-Rule, Multi-Input, or Multi-Step tiers. The online comparison similarly reports empirical drops under the Cognitive Simulator's injected perturbations; although the simulator is the weakest component (calibrated from only 10 participants and with p0, tau, K unreported), it is an input generator, not a fitted predictor of model performance, so the degradation finding is not a fitted parameter renamed as a prediction. The only self-citation (DataGovBench/DataGovAgent [24]) appears as a related-work comparison and as one of five agent baselines; it is not load-bearing for the paper's main contributions. Concerns about simulator fidelity and model-shaped selection effects are validity risks, not circular reductions.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

The benchmark depends on several assumptions about user behavior and label correctness that are only weakly supported. The simulator's parameters are not reported, and the consensus filter is a heuristic.

free parameters (3)
  • p0 (initial perturbation probability) = not reported
    Used in Algorithm 1 for noise injection; calibrated to the 10-participant user study but numeric value not given.
  • tau (decay constant) = not reported
    Controls decay of perturbation probability over turns; not specified in the paper.
  • K (cognitive window capacity) = 3 (from Fig. 4)
    Sliding window size limiting subtasks per turn; no justification for the specific value.
axioms (4)
  • domain assumption The 10-participant user study is representative of real user behavior.
    Used to derive noise types and calibrate the Cognitive Simulator (Section III-C).
  • domain assumption The consensus filter (with human verification) ensures ground-truth correctness.
    Acceptance rule in Section IV-B-a; if wrong, ground-truth labels are biased.
  • domain assumption The Level-Accuracy metric S_i is a valid measure of partial task proficiency.
    Metric definition in Section V-A-b lacks a formal specification.
  • domain assumption DAG decomposition preserves task semantics.
    Task Decomposer in Section IV-C-a assumes any multi-step workflow can be safely decomposed into ordered subtasks.
invented entities (3)
  • Cognitive Simulator no independent evidence
    purpose: Simulates user behavior in multi-turn interactions
    Only grounded in 10-participant study; no external validation.
  • Perturbation-Resolution Cycle no independent evidence
    purpose: Injects noise and queues corrective messages
    Constructed from qualitative user logs; not independently validated.
  • Cognitive-Load Window no independent evidence
    purpose: Bounds per-turn subtask load
    Derived from the user study but no quantitative support.

pith-pipeline@v1.3.0-alltime-deepseek · 17429 in / 12186 out tokens · 119341 ms · 2026-08-04T01:42:37.219104+00:00 · methodology

0 comments
read the original abstract

Tabular data processing is central to data work, and LLM-based assistants have recently shown promising capabilities in supporting such tasks. However, existing benchmarks primarily focus on table reasoning under single-turn, fully specified instructions, underrepresenting complex table processing that unfolds through multi-turn interactions with evolving user requirements. To bridge this gap, we introduce CITBench, a comprehensive benchmark for evaluating LLMs on interactive tabular data processing. CITBench features a comprehensive taxonomy across four high-level categories--table matching, cleaning, augmentation, and transformation--spanning 18 task types and 1,296 instances curated from datasets across diverse domains. The benchmark supports both offline and online evaluation, where the online setting models multi-turn interactions under constrained operation procedures and structured task scripts, capturing key potential behavioral characteristics of user-in-the-loop tabular data processing. We evaluate a broad suite of open-source and closed-source LLMs on CITBench, revealing a consistent trend: while current models perform well on simple tables and rules, their performance degrades significantly with increasing table complexity, tighter rule dependencies, and noisy multi-turn interaction simulations. These results highlight persistent challenges in understanding, planning, and table-structure awareness for LLMs in extended interactive data processing scenarios.

Figures

Figures reproduced from arXiv: 2608.00018 by Hao Liang, Wei Liu, Wentao Zhang, Xi Yan, Yang Gu, Zhou Liu, Zihan Nan.

Figure 1
Figure 1. Figure 1: Tabular Data Processing via Multi-turn Interaction. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Taxonomy and Dimensions of CITBench. categories of tabular data processing tasks, enabling broader evaluation beyond task-specific benchmarks (see Table I). B. Online and Multi-turn Interaction Benchmarks The evaluation of LLM agents increasingly moves from static instruction following toward online and multi-turn in￾teraction benchmarks that emphasize decision-making under feedback and evolving contexts [… view at source ↗
Figure 3
Figure 3. Figure 3: Key statistics of CITBench. A. Offline Task Synthesis The offline stratum forms the baseline dataset pool of CITBench, dedicated to curating foundational multi-step data processing structures across heterogeneous matrices. a) Data Acquisition and Table Adaptation: We curate 50 public source datasets across 10 distinct domains (e.g., Catering, Business, Technology, and Healthcare), meticulously balanced acr… view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the Offline and Online task construction pipelines in CITBench. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An example task specification for the Data Imputation category in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance of Different Models on Online Tasks. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Model Performance Across Interaction Turns Under Fixed Total [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Model Performance by Dataset Scale. c) The Impact of Interaction Turns on Performance [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Model Performance on Concat versus Online Tasks [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Dynamic Trajectory of Average Generated Tokens per Turn under [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Causes of Model Task Execution Failures D. Discussion Following our multi-dimensional assessment matrix, this section discusses the fundamental structural bottlenecks ob￾served across modern language agents and details strategic architectures to advance autonomous tabular intelligence. a) Challenge 1 - Execution Fidelity: The primary barrier to autonomous table manipulation is the lack of execution￾level … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 12 linked inside Pith

  1. [1]

    Sequential deep learning for credit risk monitoring with tabular financial data,

    J. M. Clements, D. Xu, N. Yousefi, and D. Efimov, “Sequential deep learning for credit risk monitoring with tabular financial data,”arXiv preprint arXiv:2012.15330, 2020

    Pith/arXiv arXiv 2012

  2. [2]

    Trust issues: Uncertainty estima- tion does not enable reliable ood detection on medical tabular data,

    D. Ulmer, L. Meijerink, and G. Cin `a, “Trust issues: Uncertainty estima- tion does not enable reliable ood detection on medical tabular data,” in Machine Learning for Health. PMLR, 2020, pp. 341–354

    2020

  3. [3]

    Benchmarking spreadsheet systems,

    S. Rahman, K. Mack, M. Bendre, R. Zhang, K. Karahalios, and A. Parameswaran, “Benchmarking spreadsheet systems,” inProceedings of the 2020 acm sigmod international conference on management of data, 2020, pp. 1589–1599

    2020

  4. [4]

    Overview and importance of data quality for machine learning tasks,

    A. Jain, H. Patel, L. Nagalapatti, N. Gupta, S. Mehta, S. Guttula, S. Mujumdar, S. Afzal, R. Sharma Mittal, and V . Munigala, “Overview and importance of data quality for machine learning tasks,” inProceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, 2020, pp. 3561–3562

    2020

  5. [5]

    Data mining preparation: process, techniques and major issues in data analysis,

    M. A. Jassim and S. N. Abdulwahid, “Data mining preparation: process, techniques and major issues in data analysis,” inIOP conference series: materials science and engineering, vol. 1090, no. 1. IOP Publishing, 2021, p. 012053

    2021

  6. [6]

    Large language model for table processing: A survey,

    W. Lu, J. Zhang, J. Fan, Z. Fu, Y . Chen, and X. Du, “Large language model for table processing: A survey,”Frontiers of Computer Science, vol. 19, no. 2, p. 192350, 2025

    2025

  7. [7]

    Jellyfish: A large lan- guage model for data preprocessing,

    H. Zhang, Y . Dong, C. Xiao, and M. Oyamada, “Jellyfish: A large lan- guage model for data preprocessing,”arXiv preprint arXiv:2312.01678, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Table-llm-specialist: Language model specialists for tables using iter- ative fine-tuning,

    J. Xing, Y . He, M. Zhou, H. Dong, S. Han, D. Zhang, and S. Chaudhuri, “Table-llm-specialist: Language model specialists for tables using iter- ative fine-tuning,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 35 431–35 448

    2025

  9. [9]

    Iterclean: An iterative data cleaning framework with large language models,

    W. Ni, K. Zhang, X. Miao, X. Zhao, Y . Wu, and J. Yin, “Iterclean: An iterative data cleaning framework with large language models,” in Proceedings of the ACM Turing Award Celebration Conference-China 2024, 2024, pp. 100–105

    2024

  10. [10]

    Cocoon: Semantic table profiling using large language models,

    Z. Huang and E. Wu, “Cocoon: Semantic table profiling using large language models,” inProceedings of the 2024 Workshop on Human-In- the-Loop Data Analytics, 2024, pp. 1–7

    2024

  11. [11]

    Retclean: Retrieval-based data cleaning using llms and data lakes,

    Z. A. Naeem, M. S. Ahmad, M. Eltabakh, M. Ouzzani, and N. Tang, “Retclean: Retrieval-based data cleaning using llms and data lakes,” Proceedings of the VLDB Endowment, vol. 17, no. 12, pp. 4421–4424, 2024

    2024

  12. [12]

    A zero-training error correction system with large language models,

    Y . Wu, C. Yang, M. Zhu, X. Miao, W. Ni, M. Xi, X. Zhao, and J. Yin, “A zero-training error correction system with large language models,” in 2025 IEEE 41st International Conference on Data Engineering (ICDE), 2025, pp. 2949–2962

    2025

  13. [13]

    Tap4llm: Table provider on sampling, augmenting, and packing semi-structured data for large language model reasoning,

    Y . Sui, J. Zou, M. Zhou, X. He, L. Du, S. Han, and D. Zhang, “Tap4llm: Table provider on sampling, augmenting, and packing semi-structured data for large language model reasoning,” inFindings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 10 306–10 323

    2024

  14. [14]

    Table-gpt: Table fine-tuned gpt for diverse table tasks,

    P. Li, Y . He, D. Yashar, W. Cui, S. Ge, H. Zhang, D. Rifinski Fainman, D. Zhang, and S. Chaudhuri, “Table-gpt: Table fine-tuned gpt for diverse table tasks,”Proceedings of the ACM on Management of Data, vol. 2, no. 3, pp. 1–28, 2024

    2024

  15. [15]

    Tablellm: Enabling tabular data manipulation by llms in real office usage scenarios,

    X. Zhang, S. Luo, B. Zhang, Z. Ma, J. Zhang, Y . Li, G. Li, Z. Yao, K. Xu, J. Zhouet al., “Tablellm: Enabling tabular data manipulation by llms in real office usage scenarios,” inFindings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 10 315–10 344

    2025

  16. [16]

    Unidm: A unified framework for data manipulation with large language models,

    Y . Qian, Y . He, R. Zhu, J. Huang, Z. Ma, H. Wang, Y . Wang, X. Sun, D. Lian, B. Dinget al., “Unidm: A unified framework for data manipulation with large language models,”Proceedings of Machine Learning and Systems, vol. 6, pp. 465–482, 2024

    2024

  17. [17]

    Clamber: A benchmark of identifying and clarifying ambiguous information needs in large language models,

    T. Zhang, P. Qin, Y . Deng, C. Huang, W. Lei, J. Liu, D. Jin, H. Liang, and T.-S. Chua, “Clamber: A benchmark of identifying and clarifying ambiguous information needs in large language models,”arXiv preprint arXiv:2405.12063, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    On over- coming miscalibrated conversational priors in llm-based chatbots,

    C. Herlihy, J. Neville, T. Schnabel, and A. Swaminathan, “On over- coming miscalibrated conversational priors in llm-based chatbots,”arXiv preprint arXiv:2406.01633, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts,

    J. D. Zamfirescu-Pereira, R. Y . Wong, B. Hartmann, and Q. Yang, “Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts,” inProceedings of the 2023 CHI conference on human factors in computing systems, 2023, pp. 1–21

    2023

  20. [20]

    Evaluating text-to-sql model failures on real-world data,

    M. Ganti, L. Orr, and S. Wu, “Evaluating text-to-sql model failures on real-world data,” in2024 IEEE 40th International Conference on Data Engineering (ICDE), 2024, pp. 1–1

    2024

  21. [21]

    Spreadsheetbench: Towards challenging real world spreadsheet manipulation,

    Z. Ma, B. Zhang, J. Zhang, J. Yu, X. Zhang, X. Zhang, S. Luo, X. Wang, and J. Tang, “Spreadsheetbench: Towards challenging real world spreadsheet manipulation,”Advances in Neural Information Pro- cessing Systems, vol. 37, pp. 94 871–94 908, 2024

    2024

  22. [22]

    Sheetagent: towards a generalist agent for spreadsheet reasoning and manipulation via large language models,

    Y . Chen, Y . Yuan, Z. Zhang, Y . Zheng, J. Liu, F. Ni, J. Hao, H. Mao, and F. Zhang, “Sheetagent: towards a generalist agent for spreadsheet reasoning and manipulation via large language models,” inProceedings of the ACM on Web Conference 2025, 2025, pp. 158–177

    2025

  23. [23]

    Autodcworkflow: Llm-based data cleaning workflow auto-generation and benchmark,

    L. Li, L. Fang, and V . I. Torvik, “Autodcworkflow: Llm-based data cleaning workflow auto-generation and benchmark,”arXiv preprint arXiv:2412.06724, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Datagovbench: Benchmarking llm agents for real-world data governance workflows,

    Z. Liu, Z. Han, G. Yan, H. Liang, B. Zeng, X. Chen, Y . Song, and W. Zhang, “Datagovbench: Benchmarking llm agents for real-world data governance workflows,”arXiv preprint arXiv:2512.04416, 2025

    arXiv 2025

  25. [25]

    Compositional semantic parsing on semi- structured tables,

    P. Pasupat and P. Liang, “Compositional semantic parsing on semi- structured tables,”arXiv preprint arXiv:1508.00305, 2015

    Pith/arXiv arXiv 2015

  26. [26]

    Tabfact: A large-scale dataset for table-based fact verification,

    W. Chen, H. Wang, J. Chen, Y . Zhang, H. Wang, S. Li, X. Zhou, and W. Y . Wang, “Tabfact: A large-scale dataset for table-based fact verification,”arXiv preprint arXiv:1909.02164, 2019

    Pith/arXiv arXiv 1909

  27. [27]

    Sheetcopilot: Bringing software productivity to the next level through large language models,

    H. Li, J. Su, Y . Chen, Q. Li, and Z.-X. Zhang, “Sheetcopilot: Bringing software productivity to the next level through large language models,” Advances in Neural Information Processing Systems, vol. 36, pp. 4952– 4984, 2023

    2023

  28. [28]

    Wrangler: Interactive visual specification of data transformation scripts,

    S. Kandel, A. Paepcke, J. Hellerstein, and J. Heer, “Wrangler: Interactive visual specification of data transformation scripts,” inProceedings of the sigchi conference on human factors in computing systems, 2011, pp. 3363–3372

    2011

  29. [29]

    Intercode: Stan- dardizing and benchmarking interactive coding with execution feed- back,

    J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao, “Intercode: Stan- dardizing and benchmarking interactive coding with execution feed- back,”Advances in Neural Information Processing Systems, vol. 36, pp. 23 826–23 854, 2023

    2023

  30. [30]

    Ds-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.- t. Yih, D. Fried, S. Wang, and T. Yu, “Ds-1000: A natural and reliable benchmark for data science code generation,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 18 319–18 345

    2023

  31. [31]

    Tablebench: A comprehensive and complex benchmark for table question answering,

    X. Wu, J. Yang, L. Chai, G. Zhang, J. Liu, X. Du, D. Liang, D. Shu, X. Cheng, T. Sunet al., “Tablebench: A comprehensive and complex benchmark for table question answering,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 24, 2025, pp. 25 497– 25 506

    2025

  32. [32]

    Realhitbench: A comprehensive realistic hierarchical table benchmark for evaluating llm-based table analysis,

    P. Wu, Y . Yang, G. Zhu, C. Ye, H. Gu, X. Lu, R. Xiao, B. Bao, Y . He, L. Zhaet al., “Realhitbench: A comprehensive realistic hierarchical table benchmark for evaluating llm-based table analysis,”arXiv preprint arXiv:2506.13405, 2025

    arXiv 2025

  33. [33]

    Spreadsheetcoder: Formula prediction from semi-structured context,

    X. Chen, P. Maniatis, R. Singh, C. Sutton, H. Dai, M. Lin, and D. Zhou, “Spreadsheetcoder: Formula prediction from semi-structured context,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 1661–1672

    2021

  34. [34]

    Instructexcel: A benchmark for natural language instruction in excel,

    J. Payan, S. Mishra, M. Singh, C. Negreanu, C. Poelitz, C. Baral, S. Roy, R. Chakravarthy, B. Van Durme, and E. Nouri, “Instructexcel: A benchmark for natural language instruction in excel,”arXiv preprint arXiv:2310.14495, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains,

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ-bench: A benchmark for tool-agent-user interaction in real-world domains,”arXiv preprint arXiv:2406.12045, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  37. [37]

    Dynasaur: Large language agents beyond predefined actions,

    D. Nguyen, V . D. Lai, S. Yoon, R. A. Rossi, H. Zhao, R. Zhang, P. Mathur, N. Lipka, Y . Wang, T. Buiet al., “Dynasaur: Large language agents beyond predefined actions,”arXiv preprint arXiv:2411.01747, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Codeassistbench (cab): Dataset & benchmarking for multi-turn chat-based code assis- tance,

    M. Kim, S. Garg, B. Ray, V . Kumar, and A. Deoras, “Codeassistbench (cab): Dataset & benchmarking for multi-turn chat-based code assis- tance,”arXiv preprint arXiv:2507.10646, 2025

    arXiv 2025

  39. [39]

    Ida-bench: Evaluating llms on interactive guided data analysis,

    H. Li, H. Liu, T. Zhu, T. Guo, Z. Zheng, X. Deng, and M. I. Jordan, “Ida-bench: Evaluating llms on interactive guided data analysis,”arXiv preprint arXiv:2505.18223, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Learning to ask: When llm agents meet unclear instruction,

    W. Wang, S. Juluan, Z. Ling, Y .-K. Chan, C. Wang, C. Lee, Y . Yuan, J.-t. Huang, W. Jiao, and M. R. Lyu, “Learning to ask: When llm agents meet unclear instruction,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 21 784– 21 795

    2025

  41. [41]

    Clarifymt-bench: Benchmarking and improving multi-turn clarification for conversational large language models,

    S. Luo, Y . Huang, M. Li, S. Meng, F. Liu, Z. Hu, J. Feng, and Q. Liu, “Clarifymt-bench: Benchmarking and improving multi-turn clarification for conversational large language models,”arXiv preprint arXiv:2512.21120, 2025

    arXiv 2025

  42. [42]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhouet al., “Metagpt: Meta programming for a multi-agent collaborative framework,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 23 247–23 275

    2024

  43. [43]

    Camel: Communicative agents for

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “Camel: Communicative agents for” mind” exploration of large language model society,”Advances in neural information processing systems, vol. 36, pp. 51 991–52 008, 2023

    2023

  44. [44]

    Cleanagent: Automating data standard- ization with llm-based agents,

    D. Qi, Z. Miao, and J. Wang, “Cleanagent: Automating data standard- ization with llm-based agents,”arXiv preprint arXiv:2403.08291, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Chatdev: Communicative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y . Dang, J. Li, C. Yang, W. Chen, Y . Su, X. Conget al., “Chatdev: Communicative agents for software development,” inProceedings of the 62nd annual meeting of the associ- ation for computational linguistics (volume 1: Long papers), 2024, pp. 15 174–15 186

    2024