Pith. sign in

REVIEW 4 major objections 6 minor 34 references

TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces TReB, a 26-subtask benchmark with three inference modes, and reports that current LLMs still have significant room to improve on table reasoning.

desk verdict Useful benchmark artifact, but the headline ranking is internally contradicted and the unvalidated LLM judge makes the leaderboard unreliable until fixed. read the letter →

arxiv 2506.18421 v2 pith:IBY5SVFD submitted 2025-06-23 cs.CL cs.AI

Ce Li , Xiaofan Liu , Zhiyan Song , Ce Chi , Chen Zhao , Jingjing Yang , Zhendong Wang , Kexin Yang
show 4 more authors
Boshen Shi Xing Wang Chao Deng Junlan Feng
This is my paper
classification cs.CLcs.AI
keywords tablereasoningbenchmarklargelanguagemodelschain-of-thoughtprogram-of-thoughtinterleavedLLM-as-a-judgedataanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TReB is a benchmark for measuring how well large language models reason about data stored in tables, a setting common in business and industry. The paper's goal is to provide a comprehensive, objective, and multi-dimensional evaluation system: 7,790 curated question-answer pairs, 26 subtasks grouped into six core skills, and three inference modes. It reports results for 26 models and claims that existing LLMs still have significant room for improvement, especially on multi-step and calculation-heavy tasks. The authors also argue that the benchmark improves on earlier table benchmarks by combining cleaned public data, real-world web tables, and proprietary table QA data with metrics that tolerate valid paraphrases.

What carries the argument

The central mechanism is the comparison of three inference modes under one evaluation pipeline. Textual Chain-of-Thought (TCoT) has the model reason and answer in plain text; Program-of-Thought (PoT) has it write executable code in a sandbox and returns the execution result; Interleaved Chain-of-Thought (ICoT) alternates planning, code execution, and feedback so the model can correct its own mistakes. The second load-bearing component is the evaluation metric: an LLM-as-a-judge model scores semantic correctness of the final answer, supplemented by ROUGE-L for text output and exact-match accuracy for single-number answers. Together these components let the benchmark attribute model failure to textual reasoning, code generation, or lack of iterative correction.

What would settle it

Take a random sample of student answers from several subtasks, have human annotators score them on the same rubric, and compare those scores with the judge model's scores. If the judge disagrees systematically with humans on particular subtasks or phrasing styles, the reported model rankings and the ICoT-over-TCoT conclusion would not be trustworthy.

Watch

Extended reading notes

Core claim

The paper's central claim is that TReB offers a comprehensive, objective, and multi-dimensional evaluation system for table reasoning. The dataset spans six core skills—natural language understanding, table understanding, basic table operations, computational operations, data analysis, and advanced multi-step analysis—and the evaluation framework scores eligible tasks under three inference modes. The headline empirical finding is that existing LLMs still have significant room for improvement: the best model reaches 69.51 on the judge-based overall metric, and exact-match accuracy on numeric answers tops out at 59.66. The authors take these results as evidence that the benchmark can support fair, reproducible comparisons and reveal where current models fall short.

Load-bearing premise

The load-bearing premise is that the LLM-as-a-judge scorer, used without being validated against human ratings on this benchmark, produces unbiased and precise scores; the paper acknowledges residual judge bias as a limitation in Section 6.1.

Editorial extensions

If this is right

  • A publicly available 26-subtask benchmark makes table-reasoning results comparable across models, so a new model can be placed on the same ranking rather than being measured on an idiosyncratic task set.
  • ICoT's advantage on multi-step and calculation-heavy tasks suggests that practical table-reasoning systems should let the model write code, execute it, receive feedback, and revise.
  • The low exact-match ceiling on numeric answers indicates that current LLMs cannot be trusted for precise table arithmetic without programmatic support.
  • Reporting both ROUGE-L and judge-based scores matters because they mostly agree on rankings but diverge on individual valid answers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would validate the judge metric against human ratings on a sample of TReB items; unless agreement is high, the reported ordering could reflect judge preferences rather than table-reasoning skill.
  • The six-skill hierarchy suggests a cheap diagnostic: models that pass table understanding but fail advanced analysis have a multi-step reasoning bottleneck, while models that fail all skills may not parse tables reliably at all.
  • Because PoT and ICoT hide the table from the model until it loads it by code, performance on large tables may depend as much on file-handling and data-frame skills as on reasoning; adding a mode that exposes the same table as text would separate the two.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces TReB, a table-reasoning benchmark comprising 7,790 samples across 26 subtasks organized under six core skills (NLU, TU, TBO, TCO, DA, ADA), with three inference modes (TCoT, PoT, ICoT) and multiple evaluation metrics (ROUGE-L, LLM-as-a-judge, exact match accuracy). The authors describe a multi-source data collection pipeline, rule-based and LLM-based generation, a multi-stage cleaning procedure with LLM voting and manual review, and benchmark results for 26 LLMs. The central claim is that TReB provides a comprehensive, objective, multi-dimensional evaluation system and that current LLMs still have substantial room for improvement on table reasoning.

Significance. If the benchmark and its evaluation protocol are trustworthy, TReB would be a useful community resource: it is openly released, covers a wider range of subtasks than most prior table benchmarks, and explicitly supports three inference modes, which is valuable for comparing reasoning paradigms. The inclusion of exact-match accuracy for numeric answers and the emphasis on cleaning real-world tables are also positive design choices. However, the paper's empirical conclusions — in particular the model leaderboard and the ICoT-versus-TCoT comparisons — rest on assumptions that are not yet established: the LLM-as-a-judge metric is not validated against human ratings, the ground-truth cleaning pipeline can replace original answers with LLM consensus, and several subtasks contain very few samples. The reported rankings therefore should be treated as preliminary until these issues are addressed.

major comments (4)
  1. [Section 4.2.1 and Table 3] The text states that 'QwQ-32B achieves the highest overall score,' but Table 3 reports Qwen3-32B with an overall LLM-as-a-judge score of 69.97 versus 69.51 for QwQ-32B. This internal contradiction directly affects the paper's headline leaderboard claim and must be resolved, either by correcting the text or by explaining why QwQ-32B is considered the top model despite the lower reported score.
  2. [Section 3.4.2 and Section 6.1] The LLM-as-a-judge metric uses Qwen2-72B-Instruct as the sole judge and is never validated against human ratings on any TReB subtask; the paper only cites RLHF alignment and shows two cherry-picked examples in Figure 3. Since the judge is from the Qwen family and many evaluated models are also Qwen-family or Qwen-distilled models, systematic judge bias could reorder the leaderboard. The authors should report human-judge agreement (e.g., Cohen's kappa or accuracy on a stratified sample) per subtask, or at minimum evaluate with multiple judge models and show rank stability.
  3. [Section 2.4, 'QA Pair Cleaning'] The cleaning pipeline replaces the original ground-truth answer with the consensus of three candidate LLMs whenever all three agree but differ from the original answer. This means a portion of the benchmark labels are derived from model outputs, and the fraction of replaced answers is not reported. Because 'High-Quality' status can be assigned through unanimous LLM agreement without expert review, the evaluation may be biased toward what LLMs produce. The authors should quantify how many QA pairs were replaced, report expert-review results on those cases, and analyze whether measured accuracies change when only expert-confirmed labels are used.
  4. [Table 1 and Tables 2, 3, 5] Several subtasks have very small sample sizes — Table Plausibility Verification (15), Table Outlier Detection (43), Table Hypothesis Testing (42), Multi-step Conditional Calculation (17) — yet no error bars, confidence intervals, or significance tests are reported anywhere in Tables 2, 3, or 5. Without such information, small overall-score differences such as 69.97 versus 69.51 cannot be distinguished from sampling noise, and the per-subtask conclusions drawn in Section 4.2 are not statistically grounded. The authors should report per-subtask sample sizes and either provide bootstrap confidence intervals or explicitly refrain from ranking models on subtasks with very small n.
minor comments (6)
  1. [Abstract] The abstract says 'table reasoning evolution benchmark'; this should read 'evaluation benchmark'.
  2. [Section 2.4 and Section 2.5] Section 2.4 reports '2.75 million high-quality data,' while Section 2.5 says the final evaluation dataset comprises 7,790 samples; the relationship between these numbers should be clarified so readers understand how the large cleaned corpus maps to the final benchmark.
  3. [Section 4.1.2] The statement that the experiments result in '1,794 experimental groups' is not obviously consistent with 26 models, 26 tasks, and up to three inference modes; the computation should be explained.
  4. [Section 3.3.3] The phrase 'compared to ToT and PoT' should be 'compared to TCoT and PoT,' since ToT (Tree-of-Thoughts) is not one of the three modes introduced in this paper.
  5. [Section 4.2.2] The claim that ROUGE-L and LLM-as-a-judge rankings are 'highly similar' is not supported by a rank-correlation coefficient or any quantitative comparison; adding Spearman's rho would make this claim testable.
  6. [Figure 2] The left panel labels the comparison as covering 'all tasks,' but NLU is excluded because it uses TCoT only; the caption should state this explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TReB is a benchmark-and-evaluation paper whose ground truth is anchored in external datasets, deterministic rules, and human review, and whose LLM-judge metric is a validity limitation rather than a by-construction reduction.

full rationale

TReB does not derive a result from an input in a way that makes the output equivalent to the input. The dataset is assembled from external public corpora (Section 2.2: MMLU, GSM8K, AIT-QA, ToTTo, HybridQA, etc.), deterministic rule-based generation (Section 2.3), and manual expert review (Section 2.4: 'Manual Review Intervention. Domain experts re-annotate these cases'; Section 2.5: 'Each instance undergoes manual annotation and a dual quality-control process'). The evaluation metric in question, LLM-as-a-judge with Qwen2-72B-Instruct, is an external model explicitly excluded from the evaluated set (Section 4.1.2), and the paper also reports exact-match accuracy on numeric tasks (Section 4.2.5, Table 5), which does not depend on the judge. The framework's effectiveness claims are empirical (cross-mode score comparisons in Figure 2; a code-execution case study in Figure 4), not reduction-by-construction. The acknowledged limitation in Section 6.1 that LLM-as-a-judge 'may inadvertently introduce biases' is a validity threat to rankings, but it is disclosed and does not make the rankings equivalent to the benchmark's inputs by definition. There are no load-bearing self-citations or imported uniqueness theorems. The internal inconsistency between the Section 4.2.1 claim that QwQ-32B achieves the highest overall score and Table 3 showing Qwen3-32B at 69.97 versus QwQ-32B at 69.51 is a reporting/correctness problem, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The entries below are design choices the benchmark depends on. They are not fitted to a target result, but they shape the dataset and scoring, so they are part of what the central claim rests on.

free parameters (3)
  • table cell character limit = 30000
    Chosen threshold in Table Cleaning (Section 2.4) excluding larger tables from the benchmark; affects which real-world tables are represented.
  • maximum empty cell proportion = 0.7
    Chosen threshold in Table Cleaning (Section 2.4) to ensure usability; hand-set and influences table selection.
  • number of candidate LLMs in QA cleaning = 3
    Three candidate LLMs are used for voting (Section 2.4); hand-selected and shapes ground-truth revisions.
assumptions (4)
  • domain assumption The six core skills and 26 subtasks cover the full spectrum of table reasoning abilities.
    Proposed in Section 2.1; completeness is asserted rather than demonstrated against an external definition of table reasoning.
  • domain assumption LLM-as-a-judge scores approximate human judgment of answer correctness.
    Section 3.4.2 uses Qwen2-72B-Instruct as a surrogate for human evaluation; no human validation is reported for this benchmark.
  • domain assumption Tables with a single-row header, at most 30,000 characters, and at most 70% empty cells are representative of real-world tables for evaluation.
    These cleaning criteria in Section 2.4 define which tables enter the benchmark and therefore limit what TReB measures.
  • domain assumption General natural language datasets are valid components of a table reasoning benchmark.
    Section 2.2.1 includes MMLU, GSM8K, and other non-table datasets under the NLU core skill.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models." pith.science (2026). https://pith.science/paper/IBY5SVFD

@misc{pith2026250618421,
  author       = {Pith},
  title        = {Pith review of: TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IBY5SVFD}},
  note         = {Machine review of arXiv:2506.18421}
}
read the original abstract

The majority of data in businesses and industries is stored in tables, databases, and data warehouses. Reasoning with table-structured data poses significant challenges for large language models (LLMs) due to its hidden semantics, inherent complexity, and structured nature. One of these challenges is lacking an effective evaluation benchmark fairly reflecting the performances of LLMs on broad table reasoning abilities. In this paper, we fill in this gap, presenting a comprehensive table reasoning evolution benchmark, TReB, which measures both shallow table understanding abilities and deep table reasoning abilities, a total of 26 sub-tasks. We construct a high quality dataset through an iterative data processing procedure. We create an evaluation framework to robustly measure table reasoning capabilities with three distinct inference modes, TCoT, PoT and ICoT. Further, we benchmark over 20 state-of-the-art LLMs using this frame work and prove its effectiveness. Experimental results reveal that existing LLMs still have significant room for improvement in addressing the complex and real world Table related tasks. Both the dataset and evaluation framework are publicly available, with the dataset hosted on huggingface.co/datasets/JT-LM/JIUTIAN-TReB and the framework on github.com/JT-LM/jiutian-treb.

Figures

Figures reproduced from arXiv: 2506.18421 by the authors.

Figure 1
Figure 1. Evaluation Framework Overview. A robust framework is specifically designed with organized data, three inference [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Comparison of table reasoning capability under different reasoning modes. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Two representative cases that highlight the differences between various evaluation metrics. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A representative case that demonstrates the differences between different reasoning modes. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 7 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [5]

    Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588,

    arXiv

  3. [6]

    R., et al

    Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Lang- don, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B. R., et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3697–3711,

    work page 2021

  4. [7]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  5. [10]

    Rlhf workflow: From reward modeling to online rlhf

    Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y ., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863,

    arXiv

  6. [11]

    The llama 3 herd of models

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  7. [12]

    A survey on llm-as-a- judge

    Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y ., Ma, S., Liu, H., et al. A survey on llm-as-a- judge. arXiv preprint arXiv:2411.15594,

    arXiv

  8. [13]

    K., Luo, F., Xiong, Y ., and Liang, W

    Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y ., Li, Y . K., Luo, F., Xiong, Y ., and Liang, W. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. arXiv preprint arXiv:2401.14196,

    arXiv

Show all 34 references
  1. [14]

    Measuring math- ematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

    arXiv

  2. [15]

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186,

    arXiv

  3. [17]

    Followeval: A multi-dimensional bench- mark for assessing the instruction-following capability of large language models

    Jing, Y ., Jin, R., Hu, J., Qiu, H., Wang, X., Wang, P., and Xiong, D. Followeval: A multi-dimensional bench- mark for assessing the instruction-following capability of large language models. arXiv preprint arXiv:2311.09829,

    arXiv

  4. [18]

    Ait-qa: Question answering dataset over complex tables in the airline industry

    Katsis, Y ., Chemmengath, S., Kumar, V ., Bharadwaj, S., Canim, M., Glass, M., Gliozzo, A., Pan, F., Sen, J., Sankaranarayanan, K., et al. Ait-qa: Question answering dataset over complex tables in the airline industry. InPro- ceedings of the 2022 Conference of the North Americ...

    2022

  5. [20]

    Kim, Y ., Yim, M., and Song, K. Y . Tablevqa-bench: A visual question answering benchmark on multiple table domains. arXiv preprint arXiv:2404.19205,

    arXiv

  6. [21]

    Tableqakit: a com- prehensive and practical toolkit for table-based question answering

    Lei, F., Luo, T., Yang, P., Liu, W., Liu, H., Lei, J., Huang, Y ., Wei, Y ., He, S., Zhao, J., et al. Tableqakit: a com- prehensive and practical toolkit for table-based question answering. arXiv preprint arXiv:2310.15075,

    arXiv

  7. [22]

    Uhgeval: Benchmarking the hallucination of chinese large language models via unconstrained generation

    Liang, X., Song, S., Niu, S., Li, Z., Xiong, F., Tang, B., Wang, Y ., He, D., Cheng, P., Wang, Z., et al. Uhgeval: Benchmarking the hallucination of chinese large language models via unconstrained generation. arXiv preprint arXiv:2311.15296,

    arXiv

  8. [26]

    Deepseekmath: Push- ing the limits of mathematical reasoning in open language models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  9. [27]

    Tablegpt2: A large multimodal model with tabular data integration

    Su, A., Wang, A., Ye, C., Zhou, C., Zhang, G., Chen, G., Zhu, G., Wang, H., Xu, H., Chen, H., et al. Tablegpt2: A large multimodal model with tabular data integration. arXiv preprint arXiv:2411.02059,

    arXiv

  10. [28]

    H., and Li, B

    Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y ., Gao, J., Awadallah, A. H., and Li, B. Adversarial glue: A multi- task benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840,

    arXiv

  11. [29]

    Mac-sql: A multi-agent collaborative framework for text-to-sql.arXiv preprint arXiv:2312.11242,

    Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.-W., Yin, D., Sun, X., et al. Mac-sql: A multi-agent collaborative framework for text-to-sql.arXiv preprint arXiv:2312.11242,

    arXiv

  12. [30]

    D., Sung, F., Vinyes, M., Ying, Z., Zhu, Z., et al

    Wang, H., Unsal, M., Lin, X., Baksys, M., Liu, J., Santos, M. D., Sung, F., Vinyes, M., Ying, Z., Zhu, Z., et al. Kimina-prover preview: Towards large formal reason- ing models with reinforcement learning. arXiv preprint arXiv:2504.11354,

    arXiv

  13. [31]

    Mmqa: Evaluating llms with multi-table multi-hop complex questions

    Wu, J., Yang, L., Li, D., Ji, Y ., Okumura, M., and Zhang, Y . Mmqa: Evaluating llms with multi-table multi-hop complex questions. In The Thirteenth International Con- ference on Learning Representations (ICLR), 2025a. Wu, X., Yang, J., Chai, L., Zhang, G., Liu, J., Du, X., Li...

    arXiv

  14. [32]

    arXiv preprint arXiv:2403.04652,

    ai. arXiv preprint arXiv:2403.04652,

    arXiv

  15. [33]

    Spider: A large-scale human-labeled dataset for complex and cross- domain semantic parsing and text-to-sql task

    Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. Spider: A large-scale human-labeled dataset for complex and cross- domain semantic parsing and text-to-sql task. In Proceed- ings of the 2018 Conference on Empirical Methods...

    2018

  16. [34]

    Y ., Vu, M

    Zhuo, T. Y ., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877,

    arXiv

  17. [2002]

    Totto: A controlled table- to-text generation dataset

    Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., and Das, D. Totto: A controlled table- to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1173–1186,

    2020

  18. [2004]

    Math- bench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark.arXiv preprint arXiv:2405.12209,

    Liu, H., Zheng, Z., Qiao, Y ., Duan, H., Fei, Z., Zhou, F., Zhang, W., Zhang, S., Lin, D., and Chen, K. Math- bench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark.arXiv preprint arXiv:2405.12209,

    arXiv

  19. [2015]

    Tqa-bench: Evaluating llms for multi-table question answering with scalable context and symbolic extension

    Qiu, Z., Peng, Y ., He, G., Yuan, B., and Wang, C. Tqa-bench: Evaluating llms for multi-table question answering with scalable context and symbolic extension. arXiv preprint arXiv:2411.19504,

    arXiv

  20. [2017]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  21. [2020]

    Mceval: Massively multilingual code evaluation

    Chai, L., Liu, S., Yang, J., Yin, Y ., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436,

    arXiv

  22. [2021]

    Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  23. [2022]

    J., Lefebvre, F., Brison, G., Perez-Lebel, A., and Varoquaux, G

    Kim, M. J., Lefebvre, F., Brison, G., Perez-Lebel, A., and Varoquaux, G. Table foundation models: on knowl- edge pre-training for tabular learning. arXiv preprint arXiv:2505.14415,

    arXiv

  24. [2023]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  25. [2024]

    Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., and Wang, W. Y . Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1026–1036,

    2020

  26. [2025]

    Tables as texts or images: Evaluating the table reasoning ability of llms and mllms

    16 TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models Deng, N., Sun, Z., He, R., Sikka, A., Chen, Y ., Ma, L., Zhang, Y ., and Mihalcea, R. Tables as texts or images: Evaluating the table reasoning ability of llms and mllms. In...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.