Pith. sign in

REVIEW 4 major objections 6 minor 36 references

MiMoTable: A Multi-scale Spreadsheet Benchmark with Meta Operations for Table Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper builds a real-world spreadsheet benchmark, MiMoTable, and claims that the best LLM gets 77.4% accuracy, with a new meta-operation difficulty score that explains why harder benchmarks yield lower accuracy.

desk verdict Useful new bilingual spreadsheet benchmark; the meta-operation difficulty criterion is a good idea but the 'proven effective' claim overreaches its evidence. read the letter →

arxiv 2412.11711 v2 pith:CSAVKGOE submitted 2024-12-16 cs.CL

Zheng Li , Yang Du , Mao Zheng , Mingyang Song This is my paper · ORCID
classification cs.CL
keywords spreadsheetreasoningtablebenchmarklargelanguagemodelsmetaoperationsdifficultyscoringreal-worldspreadsheetsquestionansweringmanipulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to close the gap between table-reasoning benchmarks and real-world spreadsheet use by introducing MiMoTable, a dataset of 428 genuine Excel files and 1,719 question-answer pairs spanning seven domains, both languages, and four table-reasoning tasks. It also proposes a new way to measure question difficulty: each question is tagged with one or more of six meta operations (Lookup, Edit, Compare, Calculate, Visualize, Reasoning), and a formula converts the tags into a difficulty score. The paper's key evidence is that the best tested LLM, Claude-3.5-Sonnet, scores only 77.4% accuracy on MiMoTable, and that on existing benchmarks, model accuracy falls as the meta-operation difficulty score rises (WikiSQL 1.5, WikiTableQuestion 2.0, MiMoTable-Simple 2.2). If this holds, the meta-operation score becomes a practical tool for predicting LLM performance on any table dataset before running a single model, and MiMoTable stands as a harder, more realistic test bed.

What carries the argument

The central object is the taxonomy of six meta operations, each with a difficulty grade: Lookup and Edit are grade 1, Compare, Calculate, and Visualize are grade 2, and Reasoning is grade 3. A question's difficulty score is computed as $q_{si} = m_{s_i} + \frac{\sum_{k=1}^{K_i} s_k - m_{s_i}}{M_{m_{s_i}}}$, where $m_{s_i}$ is the maximum grade among the question's operations and $M_{m_{s_i}}$ caps the sum of the remaining grades (1 for grade 1, 6 for grade 2, 8 for grade 3), giving a score in [1,4]. A dataset's difficulty is the average of its question scores. This score is what lets the paper grade existing benchmarks like WikiSQL and WikiTableQuestions on the same scale and compare them against measured LLM accuracy.

What would settle it

Hand-label a random sample of 200 questions from WikiSQL and WikiTableQuestions into the six meta operations and recompute the dataset difficulty scores; if the scores no longer sit at 1.5 and 2.0, or the accuracy drop in Figure 10 disappears with human-verified labels, the criterion's predictive claim fails.

Watch

Extended reading notes

Core claim

The paper proposes that current table-reasoning benchmarks miss real-world spreadsheet complexity, and that a new meta-operation difficulty criterion can place benchmarks from different tasks on a common scale. It validates this by showing that LLM accuracy declines monotonically as the meta-operation difficulty score rises from WikiSQL (1.5) to WikiTableQuestions (2.0) to MiMoTable-Simple (2.2), and that the best model, Claude-3.5-Sonnet, reaches only 77.4% on MiMoTable, with performance dropping to 63.3% on Reasoning questions and 72.1% on hard tables.

Load-bearing premise

The evidence that the difficulty criterion works assumes that GPT-4o's meta-operation labels for existing benchmarks are correct, and the paper reports no human check of those labels.

Editorial extensions

If this is right

  • MiMoTable provides a more realistic stress test for LLMs, since it includes real spreadsheets with complex headers, multiple sheets, multiple files, and multiple tables per sheet.
  • The meta-operation difficulty score gives a common scale for comparing benchmarks across different table tasks, so a score of 2.0 on one dataset can be interpreted as equally hard as the same score on another.
  • Model rankings on WikiSQL and WikiTableQuestions can be explained by difficulty rather than by dataset-specific quirks, and the same reasoning can be applied to any new table benchmark.
  • The 77.4% ceiling quantifies remaining headroom, and the per-operation scores identify which capabilities need the most work, such as Reasoning at 63.3% for the best model.
  • For simple and medium tables, a code-interpreter approach beats a text-based approach on Calculate and Compare, while text-based is better on hard tables and on Lookup and Reasoning, suggesting future systems should route by table difficulty and operation type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A human-annotation study of the meta-operation labels on WikiSQL and WikiTableQuestions would test whether the 1.5 and 2.0 difficulty scores are stable; if labels shift, the claimed monotonic trend may be a GPT-4o artifact.
  • The meta-operation taxonomy could be used to build difficulty-controlled subsets of a benchmark, letting researchers evaluate models on a curriculum from Lookup-only to Reasoning-heavy questions.
  • The same scoring scheme could be extended to other structured-data formats, such as databases or forms, to create comparable difficulty metrics across data modalities.
  • The paper's limitation section notes no supervised fine-tuning was performed; fine-tuning models per meta operation would reveal which operations are learnable and whether 77.4% is a data ceiling or a model ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MiMoTable, a bilingual (Chinese/English) spreadsheet reasoning benchmark built from 428 real-world spreadsheets and 1,719 (spreadsheet, question, answer) triplets. The benchmark covers four table-reasoning task types and a wide range of table structures, including simple and complex headers, multiple sheets, multiple files, and multiple tables per sheet. The authors also propose a new difficulty criterion based on six meta operations (Lookup, Edit, Calculate, Compare, Visualize, Reasoning), assign each operation a grade, and derive question-level and dataset-level difficulty scores. They evaluate 16 LLMs and report that Claude-3.5-Sonnet achieves the best accuracy of 77.4%. Finally, they grade WikiSQL and WikiTableQuestion with the same criterion and claim that the resulting difficulty ordering is validated by the observed drop in LLM accuracy across the three datasets.

Significance. If the benchmark and the difficulty criterion are validated, MiMoTable is a useful resource: it targets real-world spreadsheet complexity that most existing table benchmarks lack, it spans multiple task types and languages, and it is open-sourced. The construction pipeline is described in detail, answers were human-annotated with a reported Cohen's Kappa of 0.83, and the best-model accuracy of 77.4% indicates headroom for current LLMs. The meta-operation difficulty criterion is an interesting and potentially generalizable analytical tool. However, the evidence for the criterion's effectiveness is currently much weaker than the abstract's 'proven effectiveness' claim: the external validation rests on three dataset-level points with model-generated difficulty labels, and the within-benchmark difficulty results are not controlled for table difficulty. These issues are addressable and do not undermine the value of the dataset itself.

major comments (4)
  1. [§3.2, Figure 10] The claim that the meta-operation difficulty criterion is 'proven effective' rests on Figure 10, which shows a monotonic accuracy drop across only three datasets: WikiSQL (difficulty 1.5), WikiTableQuestion (2.0), and MiMoTable-Simple (2.2). The x-axis values are computed by applying the GPT-4o classifier in Table 8 to existing benchmarks, with no reported human validation of those classifications. Because the same model family is later evaluated, the difficulty labels and model accuracies are not independent. Moreover, the three datasets differ in answer format, table provenance, and question distribution, so the observed drop could reflect these confounds rather than the criterion's validity. Please provide human-validated difficulty labels and a controlled comparison, for example by scoring questions from a single dataset or by matching questions across table-difficulty strata.
  2. [§3.2, Table 6] Table 6 reports accuracy by question-difficulty bins without stratifying by table difficulty. For Claude-3.5-Sonnet, accuracy falls from 89.0% on questions with difficulty [1,2) to 63.3% on [3,4], while accuracy on hard tables is 72.1%. If hard tables are overrepresented among high-difficulty questions, the drop could be explained by table structure rather than by meta-operation complexity. Please report accuracy within table-difficulty strata, or a regression with table difficulty as a covariate, to support the claim that question difficulty as defined by meta operations is the driving factor.
  3. [§3.1, Table 10] All accuracy numbers in Table 6 depend on GPT-4o judging whether candidate answers are correct. The prompt in Table 10 allows a 0-1 score for open-ended questions, but the paper does not state how these scores are converted into binary accuracy, nor does it report agreement between the GPT-4o judge and human annotators. Since GPT-4o also generated the questions and is one of the evaluated models, this is a potential source of bias. Please specify the conversion rule and report a human-validated sample of judge decisions, for example at least 100 items across models and difficulty levels.
  4. [§3.2, Table 6] The reported 77.4% accuracy is computed only on questions without the Edit and Visualize meta operations, because most evaluated models cannot produce edited files or charts. The paper does not state how many of the 1,719 questions are excluded. The headline result therefore does not characterize performance on the full benchmark. Please report the size of the evaluated subset, the accuracy of GPT-4o-CI on the full set, and, if possible, an evaluation of the full benchmark using format-appropriate metrics for file and chart outputs.
minor comments (6)
  1. [Abstract and §3.2] The phrase 'proving the effectiveness' overstates the evidence presented; consider 'providing evidence for' or 'supporting' until the validation concerns are addressed.
  2. [§2.2, Eq. (3)] Equation (3) uses M_msqi before its values are specified in Eq. (5); reorder the presentation and state the assumption that each meta operation can appear at most once in a question before introducing the formula.
  3. [Table 6] For every model, the accuracy under 'Question Difficulty [1,2)' is identical to the accuracy under 'Lookup'; this suggests the two subsets coincide or that a column is mislabeled, so please clarify the subset definitions.
  4. [§2.3] The multi-file construction is described as 'randomly sampled' groups of 2-5 files, but the paper does not report how many such groups were created or whether files are reused; please add this statistic for reproducibility and to rule out cross-question leakage.
  5. [§1] The term 'multi-scale' in the benchmark name is not defined; please state explicitly that it refers to variation in sheets, files, tables per sheet, and header complexity.
  6. [Figure 7] The radar chart overlays many models and is difficult to read in print; consider separate panels or a table for the per-meta-operation accuracies.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild circularity in the MiMoTable-Simple validation point, but the meta-operation criterion itself is not fitted to accuracy and the central correlation retains independent content.

  1. other [Section 2.3 Question Generation and Section 3.2 Effectiveness of Meta Operations / Figure 10]
    "To ensure the diversity of questions, the multiple questions should contain different combinations of meta operations. ... The questions involving simple tables in MiMoTable, denoted as MiMoTable-Simple, have a difficulty of 2.2."

    MiMoTable questions were generated so that each question would contain requested meta-operation combinations, and the same meta-operation labels determine the difficulty score used for MiMoTable-Simple in the validation plot. Therefore the MiMoTable-Simple data point is not an independent test of the meta-operation criterion: its x-coordinate is an input to data construction rather than an external measurement. The other two points (WikiSQL and WikiTableQuestion) are externally classified and the accuracy decline is empirical, so the circularity is partial and does not force the main result by construction.

full rationale

The central derivation is not circular: the meta-operation difficulty scores come from a hand-defined rubric (Table 2) with grades 1-3, combined by Equation 3 and averaged by Equation 6, none of which are fitted to model accuracy. The claim that accuracy decreases as difficulty increases is an empirical correlation across WikiSQL, WikiTableQuestion, and MiMoTable-Simple. The mild circularity is that MiMoTable-Simple's difficulty score is computed from the same meta-operation taxonomy that guided its question generation, so that point does not independently validate the taxonomy. The difficulty labels for WikiSQL and WikiTableQuestion are produced by GPT-4o using the prompt in Table 8, and GPT-4o is also one of the evaluated models; this is a label-quality and potential-bias concern rather than a logical circularity. The only self-citations (Song et al., 2024a,b) appear in Section 6 as inspiration for future long-context and in-context-learning directions and are not load-bearing. The paper's overstrong wording 'proving the effectiveness' is a correctness-risk issue, not a circular derivation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The framework rests on hand-assigned operation grades and on LLM-generated labels being trustworthy; neither has independent external grounding. No new physical or causal entities are introduced, so the invented entities list is empty.

free parameters (1)
  • Meta operation grade assignments = Lookup=1, Edit=1, Compare=2, Calculate=2, Visualize=2, Reasoning=3
    Hand-chosen in Table 2 based on 'common criteria'. These grades determine every question and dataset difficulty score, so the correlation claim in Figure 10 depends on them.
assumptions (4)
  • ad hoc to paper The six meta operations are exhaustive and non-overlapping enough to represent all table-reasoning problems.
    Stated in Section 2.2 and Figure 3; the paper gives examples but no formal proof that the taxonomy covers all tasks.
  • domain assumption GPT-4o's meta-operation classifications are accurate enough to grade existing benchmarks.
    In Section 3.1, the difficulty of WikiSQL and WikiTableQuestion is computed from GPT-4o classifications, with no human agreement rate reported for these labels.
  • domain assumption GPT-4o answer judging reliably matches human judgments of correctness.
    In Section 3.1 Metrics, correctness is judged by GPT-4o, including a 0-1 score for open-ended questions, but no comparison to human judgments is reported.
  • domain assumption Spreadsheets collected from Baidu Wenku and Google searches are representative of real-world spreadsheet use.
    In Section 2.3 Table Collection, sources are described but sampling is not randomized and filtering is manual, so representativeness is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MiMoTable: A Multi-scale Spreadsheet Benchmark with Meta Operations for Table Reasoning." pith.science (2026). https://pith.science/paper/CSAVKGOE

@misc{pith2026241211711,
  author       = {Pith},
  title        = {Pith review of: MiMoTable: A Multi-scale Spreadsheet Benchmark with Meta Operations for Table Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CSAVKGOE}},
  note         = {Machine review of arXiv:2412.11711}
}
read the original abstract

Extensive research has been conducted to explore the capability of Large Language Models (LLMs) for table reasoning and has significantly improved the performance on existing benchmarks. However, tables and user questions in real-world applications are more complex and diverse, presenting an unignorable gap compared to the existing benchmarks. To fill the gap, we propose a \textbf{M}ult\textbf{i}-scale spreadsheet benchmark with \textbf{M}eta \textbf{o}perations for \textbf{Table} reasoning, named as MiMoTable. Specifically, MiMoTable incorporates two key features. First, the tables in MiMoTable are all spreadsheets used in real-world scenarios, which cover seven domains and contain different types. Second, we define a new criterion with six categories of meta operations for measuring the difficulty of each question in MiMoTable, simultaneously as a new perspective for measuring the difficulty of the existing benchmarks. Experimental results show that Claude-3.5-Sonnet achieves the best performance with 77.4\% accuracy, indicating that there is still significant room to improve for LLMs on MiMoTable. Furthermore, we grade the difficulty of existing benchmarks according to our new criteria. Experiments have shown that the performance of LLMs decreases as the difficulty of benchmarks increases, thereby proving the effectiveness of our proposed new criterion.

Figures

Figures reproduced from arXiv: 2412.11711 by the authors.

Figure 1
Figure 1. Examples of MiMoTable benchmark. To foster a comprehensive understanding of this field, researchers have proposed and developed nu￾merous table reasoning tasks, such as TableQA, Table2Text, Table Manipulation, and Advanced Data Analysis (Lu et al., 2024). Various methods have been proposed to tackle these tasks, and large language models (LLMs) have achieved promis￾ing results (Liu et al., 2022; Cheng et al., 2023).… view at source ↗
Figure 2
Figure 2. Illustrations of different table types, including simple header, complex header, single sheet, multiple [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The relationships between tasks in the existing [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The data construction pipeline of MiMoTable benchmark. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Domain distribution of all spreadsheets. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Distribution of meta operations. Difficulty of Questions. To investigate the diffi￾culty of questions, we calculate the difficulty score of each question according to the Equation 3. The score is in the range of [1, 4], so we divided the distribution of scores into thr…
Figure 7
Figure 7. Figure 7: The performance of LLMs on MiMoTable respecting to different Meta Operations, Domains and Table [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Performance on different data types. problems of spreadsheets. One approach is to repre￾sent the spreadsheet content in text form and input it into the model to directly generate answers. The other is directly using the spreadsheet as input to write code, run in a sand…
Figure 9
Figure 9. Figure 9: A spreadsheet to text with the markdown format. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: The relations between performance and the [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 7 canonical work pages

  1. [1]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv doi:10.48550/arxiv.2309.16609 2023

  2. [2]

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.91 Hybridqa: A dataset of multi-hop question answering over tabular and textual data . In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 , volume EMNLP 2020 of F...

    doi:10.18653/v1/2020.findings-emnlp.91 2020

  3. [3]

    Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian - Guang Lou, and Dongmei Zhang. 2022. https://doi.org/10.18653/V1/2022.ACL-LONG.78 Hitab: A hierarchical table dataset for question answering and natural language generation . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume...

    doi:10.18653/v1/2022.acl-long.78 2022

  4. [4]

    Smith, and Tao Yu

    Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. https://openreview.net/forum?id=lH1PV42cbF Binding language models in symbolic languages . In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwa...

    work page 2023

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur \' e lien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozi \` e...

    arXiv doi:10.48550/arxiv.2407.21783 2024

  6. [6]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. https://doi.org/10.48550/ARXIV.2401.14196 Deepseek-coder: When the large language model meets programming - the rise of code intelligence . CoRR, abs/2401.14196

    arXiv doi:10.48550/arxiv.2401.14196 2024

  7. [7]

    Xinyi He, Mengyu Zhou, Xinrun Xu, Xiaojun Ma, Rui Ding, Lun Du, Yan Gao, Ran Jia, Xu Chen, Shi Han, Zejian Yuan, and Dongmei Zhang. 2024. https://doi.org/10.1609/AAAI.V38I16.29779 Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries . In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, ...

    work page doi:10.1609/aaai.v38i16.29779 2024

  8. [8]

    Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. 2024. https://doi.org/10.48550/ARXIV.2401.05507 Infiagent-dabench: Evaluating agents on data analysis tasks . CoRR, abs/2401.05507

    arXiv doi:10.48550/arxiv.2401.05507 2024

Show all 36 references
  1. [9]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \' e lio Renard Lavaud, Marie - Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, ...

    2023 doi arXiv

  2. [10]

    Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael R

    Yannis Katsis, Saneem A. Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael R. Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, and Soumen Chakrabarti. 2022. https://doi.org/10.18653/V1/2022.NAACL-INDUSTRY.34 AIT-QA: question answe...

    2022 doi

  3. [12]

    Wang, and Tao Yu

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen - Tau Yih, Daniel Fried, Sida I. Wang, and Tao Yu. 2023. https://proceedings.mlr.press/v202/lai23b.html DS-1000: A natural and reliable benchmark for data science code generation . In Interna...

    2023

  4. [13]

    Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. 2023 a . http://papers.nips.cc/paper\_files/paper/2023/hash/0ff30c4bf31db0119a6219e0d250e037-Abstract-Conference.html Sheetcopilot: Bringing software productivity to the next level through large language models...

    2023

  5. [14]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chen - Chuan Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023 b . http://papers.nips.cc/paper\_files/paper/2023/hash...

    2023

  6. [15]

    Zheng Li, Xiang Chen, and Xiaojun Wan. 2024. https://doi.org/10.48550/ARXIV.2403.02962 Wikitableedit: A benchmark for table editing by natural language instruction . CoRR, abs/2403.02962

    2024 doi arXiv

  7. [16]

    Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian - Guang Lou. 2022. https://openreview.net/forum?id=O50443AsCP TAPEX: table pre-training via learning a neural SQL executor . In The Tenth International Conference on Learning Representations, ICLR 2...

    2022

  8. [17]

    Weizheng Lu, Jiaming Zhang, Jing Zhang, and Yueguo Chen. 2024. https://doi.org/10.48550/ARXIV.2402.05121 Large language model for table processing: A survey . CoRR, abs/2402.05121

    2024 doi arXiv

  9. [18]

    Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. 2024. https://doi.org/10.48550/ARXIV.2406.14991 Spreadsheetbench: Towards challenging real world spreadsheet manipulation . CoRR, abs/2406.14991

    2024 doi arXiv

  10. [19]

    Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \` e re, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, L \' e onard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro - Ros, Ambrose ...

    2024 doi arXiv

  11. [20]

    Kai Nakamura, Sharon Levy, Yi - Lin Tuan, Wenhu Chen, and William Yang Wang. 2022. https://doi.org/10.18653/V1/2022.FINDINGS-ACL.41 Hybridialogue: An information-seeking dialogue dataset grounded on tabular and textual data . In Findings of the Association for Computational Li...

    2022 doi

  12. [21]

    Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryscinski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, and Dragomir R. Radev. 2022. http...

    2022 doi

  13. [22]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2023 arXiv

  14. [23]

    Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das

    Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. https://doi.org/10.18653/V1/2020.EMNLP-MAIN.89 Totto: A controlled table-to-text generation dataset . In Proceedings of the 2020 Conference on Empirical Methods...

    2020 doi

  15. [24]

    Panupong Pasupat and Percy Liang. 2015. https://doi.org/10.3115/V1/P15-1142 Compositional semantic parsing on semi-structured tables . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natur...

    2015 doi

  16. [25]

    Lillicrap, Jean - Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean - Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, M...

    2024 doi arXiv

  17. [26]

    Mingyang Song, Mao Zheng, and Xuan Luo. 2024 a . https://arxiv.org/abs/2406.11629 Can many-shot in-context learning help llms as evaluators? a preliminary empirical study . Preprint, arXiv:2406.11629

    2024 arXiv

  18. [27]

    Mingyang Song, Mao Zheng, and Xuan Luo. 2024 b . https://arxiv.org/abs/2403.11802 Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models . Preprint, arXiv:2403.11802

    2024 arXiv

  19. [28]

    Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. https://doi.org/10.1145/3616855.3635752 Table meets LLM: can large language models understand structured table data? A benchmark and empirical study . In Proceedings of the 17th ACM International Conference...

    2024

  20. [29]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 doi arXiv

  21. [30]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. https://openreview.net/forum?id=WE\_vluYUL-X React: Synergizing reasoning and acting in language models . In The Eleventh International Conference on Learning Representations, ...

    2023

  22. [31]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. https://doi.org/10.18653/V1/D18-1425 Spider: A large-scale human-labeled dataset for complex and cross-domain sem...

    2018 doi

  23. [32]

    Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. 2023. https://doi.org/10.48550/ARXIV.2311.09206 Tablellama: Towards open large generalist models for tables . CoRR, abs/2311.09206

    2023 doi arXiv

  24. [33]

    Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Qingfu Zhu, and Wanxiang Che. 2024. https://doi.org/10.48550/ARXIV.2402.08259 A survey of table reasoning with large language models . CoRR, abs/2402.08259

    2024 doi arXiv

  25. [34]

    Victor Zhong, Caiming Xiong, and Richard Socher. 2017. https://arxiv.org/abs/1709.00103 Seq2sql: Generating structured queries from natural language using reinforcement learning . CoRR, abs/1709.00103

    2017 arXiv

  26. [35]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat - Seng Chua. 2021. https://doi.org/10.18653/V1/2021.ACL-LONG.254 TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance . In Proceedings of ...

    2021 doi

  27. [36]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.