Pith. sign in

REVIEW 4 major objections 6 minor 62 references

AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AraTable, a human-validated Arabic tabular benchmark, finds open-source LLMs exceed 90% on direct lookup but stay below 60% on reasoning, and introduces a self-deliberation judge that matches human scores.

desk verdict AraTable is a useful first benchmark for Arabic tabular QA, but its central findings rest on human labels and question-selection choices whose reliability is not yet shown. read the letter →

arxiv 2507.18442 v1 pith:DCEKGVSH submitted 2025-07-24 cs.CL cs.AI

Rana Alshaikh , Israa Alghanmi , Shelan Jeawak This is my paper
classification cs.CLcs.AI
keywords ArabicbenchmarktabulardatalargelanguagemodelsquestionansweringfactverificationtablereasoningLLMasjudgeself-deliberation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AraTable is a benchmark of 41 Arabic tables and 615 human-validated question-answer pairs that tests three things: direct lookup, reasoning, and fact verification. The paper's central finding is that current open-source LLMs are strong at direct extraction, usually above 90% for the best models, but drop below 60% on reasoning questions and sit in between on fact verification. The authors argue this gap is a genuine limitation in LLMs' ability to perform logical inference over Arabic structured data, not a formatting artifact, because they used a relaxed evaluation judged by native Arabic speakers. They also claim their assisted self-deliberation (ASD) evaluation, which asks two LLM judges to revisit only their disagreements, produces accuracy scores nearly identical to human judges. If true, the contribution is a reusable, publicly available Arabic table benchmark plus a cheap automated evaluation method that tracks human judgement.

What carries the argument

The load-bearing objects are the AraTable dataset itself and the assisted self-deliberation (ASD) evaluation protocol. AraTable is built by a hybrid pipeline: GPT-4 generates Arabic questions from tables converted to CSV, three native Arabic-speaking annotators verify and correct each question-answer pair, and five questions per task per table are retained, giving 615 pairs across 41 tables. ASD is an LLM-as-judge setup in which two judge models independently score each answer as True or False; when they disagree, each judge receives only the other judge's label, never its reasoning, and must justify both positions against the shared rubric before settling. The disagreement signal, not full debate, is what carries the method: it forces asymmetric self-critique with a single extra evaluation pass.

What would settle it

Re-annotate the 615 question-answer pairs and re-judge a random sample of model responses with an independent panel of native Arabic speakers; if inter-annotator agreement is low or the new panel's scores differ materially from the published ones, the below-60% reasoning result and ASD's near-perfect match to human judgement would not be reproducible.

Watch

Extended reading notes

Core claim

The paper claims that Arabic tabular understanding splits into two regimes: models excel when the answer sits in one cell, and they falter when the answer requires combining rows and columns. Across Wikipedia, real-world, and LLM-generated tables, direct QA accuracy runs above 90% for DeepSeek-V3 and close to it for Llama 3.3 and Mistral Large, while reasoning accuracy stays below 60% for every model and drops as low as 20% (Llama on real-world tables) or 14% (Jais). An Arabic-centric model, Jais 70B, performs worst overall, which the paper reads as evidence that Arabic language coverage alone does not buy tabular reasoning skill. On the evaluation side, the authors report that their ASD mechanism reduces the average absolute gap between LLM judges and human judges by roughly half, and that one judge (Qwen) matches the human accuracy scores exactly on Wikipedia and real-world datasets after deliberation.

Load-bearing premise

The entire accuracy ranking and the claimed near-identity between LLM judges and human judges rest on the correctness and consistency of the human-annotated ground truth and the nine judges' True/False labels; the paper reports no inter-annotator agreement and does not explain why only 5 of 10 GPT-4-generated questions per task were kept.

Editorial extensions

If this is right

  • AraTable gives the research community a public, human-checked yardstick for Arabic table QA, so future models can be compared on the same 615 questions.
  • The consistent reasoning shortfall, below 60% for the best model, means direct extraction should not be treated as evidence of table understanding; progress should be measured on the reasoning and fact-verification splits.
  • Real-world Arabic tables are substantially harder than Wikipedia tables, so benchmark results on tidy tables will overstate practical performance.
  • An Arabic-centric pretraining corpus is not sufficient for Arabic tabular reasoning; exposure to structured-data reasoning during training matters.
  • ASD's near-human agreement offers a low-cost path to automated evaluation, but the paper's own results show the benefit depends on the judge model, with one judge improving much more than the other.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper does not document how 5 of 10 generated questions per task were chosen, the retained set may be biased toward clearer questions; a follow-up using explicit stratified selection or releasing the discarded questions could test this.
  • ASD's alignment with humans is measured against the same rubric-dependent human labels that define ground truth; its real-world value would be better tested on answers where human judges themselves disagree.
  • The 40-row cap likely removes large-table phenomena such as long-range aggregation and scale effects; extending a subset of tables beyond this cap is a natural, testable enhancement.
  • The zero-shot-only setup probably understates what the models can do; few-shot versions of the same 615 questions are a direct next experiment the authors flag as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces AraTable, a benchmark of 615 human-validated Arabic QA pairs over 41 tables drawn from Wikipedia, real-world sources, and LLM-generated tables, spanning direct question answering, fact verification, and reasoning. The authors evaluate Llama 3.3 70B, DeepSeek-V3, Mistral Large, and Jais 70B under a zero-shot setting, using a three-round human judging procedure to define accuracy, and they propose an assisted self-deliberation (ASD) mechanism in which two LLM judges re-evaluate only their disagreements. The main empirical claim is that top models achieve high accuracy on direct table lookup (often above 90%) but fall below 60% on reasoning questions, with fact verification in between; the ASD framework is reported to align closely with human judgments after deliberation.

Significance. If the reliability concerns are addressed, AraTable fills a genuine gap: it is, to my knowledge, the first standardized Arabic tabular QA benchmark, it is publicly released, and its three-task structure (direct QA, fact verification, reasoning) follows established English-language benchmarks such as WikiTableQuestions, TabFact, and DataBench. The paper also ships the full prompts and a detailed evaluation rubric, which supports reproducibility. The ASD mechanism is a lightweight and practical contribution: it triggers self-deliberation only on disagreement and requires a single re-evaluation pass per judge. The central finding that Arabic table lookup is easier than tabular reasoning is directionally consistent with results on English tabular benchmarks, so the contribution is plausible. No circularity issue arises in validating an evaluator against human labels; the main risk is that the human labels themselves lack demonstrated reliability.

major comments (4)
  1. [§3.4, §5.1] The reliability of the human labels is not established. Three annotators filter and correct the GPT-4-generated QA pairs (§3.4), nine judges produce the final human baseline in three rounds (§5.1), and two annotators extract concise answers from Jais's verbose outputs (§4.2), yet no inter-annotator agreement statistic (e.g., Cohen's or Fleiss' kappa) is reported for any of these steps. Because every model accuracy in Tables 7–9 and every ASD gap in Tables 10–12 is computed relative to these human labels, the central direct-vs-reasoning finding is unverified without evidence that the labels are stable. Please report per-round and final agreement, describe how within-group disagreements were resolved, and add this dependency explicitly to Section 8.
  2. [§3.4] The selection of 5 of 10 generated questions per task is under-documented. The text says only that retained questions were 'selected to target information across different columns and rows,' and Prompt 3.2 explicitly instructs GPT-4 to generate 'truly complex' fact-verification questions. If the retained reasoning or fact-verification items are systematically harder than the discarded ones, the observed task gap is partly a curation artifact rather than a model property. Please document the selection protocol, report statistics comparing retained and discarded questions (e.g., answer-type balance, difficulty proxy, column/row coverage), and release the discarded pool if possible.
  3. [Tables 7–9; Table 5] Reported accuracies lack confidence intervals and significance tests. With 50 real-world reasoning questions per model (SE ≈ 7 percentage points), 25 LLM-generated reasoning questions (SE ≈ 10 points), and only 17 temporal reasoning questions in the entire benchmark, differences such as Llama 20% versus Mistral 30% on real-world reasoning (Table 8) are within plausible sampling error. Please report per-source and per-task confidence intervals or bootstrapped intervals, and temper model-hierarchy claims in Section 6.1 that are not supported by significance testing.
  4. [§5.2, Tables 10–12] The ASD validation is not yet quantitatively convincing. The claim that deliberation makes Qwen's judgments 'nearly identical' to humans is based on aggregate accuracy relative to a human baseline whose reliability is unreported, and several post-ASD entries exactly equal the human baseline (e.g., Table 10 for all five models). Please report item-level agreement between each LLM judge and the final human labels (e.g., Cohen's kappa), the number of disagreements that triggered self-deliberation, and the revision rate after deliberation. Without these numbers, the 'near-identity' and 'roughly halves' claims in Section 6.2 are not supported beyond a single illustrative example.
minor comments (6)
  1. [§5.2, Table 6] The judge model is called '4O' throughout, but reference [59] identifies it as GPT-4o; please use a consistent name and check for similar encoding artifacts elsewhere.
  2. [§4.1] Mistral Large is described as an open-source model, but it is a proprietary API model; please correct the scope statement or justify its inclusion under a broader 'publicly accessible' label.
  3. [§3.3] The text says 'For each task, GPT-4 produced 10 questions per table,' but Prompt 3.1 generates 10 reasoning and 10 non-reasoning questions in a single call, while Prompt 3.2 generates 10 fact-verification questions; please clarify the actual number of generation calls and how the 30 questions per table were assembled.
  4. [Figure 3] The caption says the figure shows 'the average unsigned error' but the y-axis label reads 'Average Disagreement Across Models'; please align the terminology and report the numeric mean |Δ| values in the text.
  5. [§6.1] The sentence 'larger token budgets and more diverse pre-training corpora ... matter more than architecture tweaks alone' is speculative and is not isolated by the experimental design; please soften or remove it.
  6. [Section 8] The limitations section should explicitly mention the absence of inter-annotator agreement analysis and the small size of the temporal reasoning subset (17 questions, Table 5).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark results and ASD alignment are empirical measurements against externally human-validated labels, not derivations from fitted inputs or self-citations.

full rationale

The paper's derivation chain is self-contained and non-circular. The central claims—model accuracies in Tables 7-9 and ASD-human alignment in Tables 10-12—are empirical measurements computed with respect to human-validated ground truth (Sections 3.4 and 5.1). No parameter is fitted to the data being 'predicted': the ASD evaluators are given the same rubric and ground-truth answers as human judges (Section 5.2), and their performance is measured by agreement with the human baseline, which is a standard external validation rather than a circular reduction. There are no load-bearing self-citations: the cited prior work (e.g., Llama, Mistral, DeepSeek, Jais, DataBench) is external, and no uniqueness theorem or ansatz from the authors' own prior work is invoked to force a choice. The observation that direct QA is easier than reasoning is partly encoded in the generation prompts (Prompt 3.1 asks for 'higher-order thinking' reasoning questions and 'straightforward' non-reasoning questions; Prompt 3.2 asks for 'truly complex' fact-verification statements), but the measured accuracy gap is not a tautology—models could in principle fail direct QA or succeed on reasoning, and the reported numbers are actual evaluation outcomes rather than consequences of definitions. The absence of inter-annotator agreement statistics and the undocumented 5-of-10 question retention criteria are reliability and correctness concerns, not circularity concerns; they do not make any result equivalent to its own input by construction. Therefore no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

This benchmark paper has no mathematical derivation, so the ledger records the design choices and domain assumptions that actually drive the reported numbers. The numeric tolerance, question retention rate, table row cap, and the ASD conflict trigger are hand-set parameters. The reliability of the human labels is the load-bearing assumption for every accuracy and agreement figure.

free parameters (4)
  • numeric tolerance = 0.005
    Rubric rule 3 (Section 5.1) treats numeric answers within plus or minus 0.005 as correct; this hand-set tolerance directly affects fact verification and numeric comparison accuracy.
  • retained questions per task per table = 5 of 10
    Section 3.4 retains five of ten GPT-4-generated questions per task after manual filtering; the undocumented selection criteria influence benchmark difficulty and all reported accuracies.
  • maximum table rows = 40
    Section 3.2 truncates tables at 40 rows to fit model input, which changes table content and question-answerability for larger original tables.
  • ASD disagreement trigger = only on conflict
    Section 5.2 invokes self-deliberation only when the two LLM judges disagree; the number of disagreement cases and the prompt template are design choices that affect the reported judge alignment.
assumptions (4)
  • domain assumption GPT-4-generated QA pairs, after human filtering, form a valid ground truth.
    Sections 3.3 and 3.4 rest the entire benchmark on this filtering process; no inter-annotator agreement is reported for the three annotators.
  • domain assumption The selected 41 Arabic tables represent meaningful Arabic tabular data.
    Section 3.1 samples Wikipedia, real-world sources, and LLM-generated tables; the paper offers no evidence of representativeness across domains, dialects, or table formats.
  • domain assumption Zero-shot prompting without format constraints is a fair evaluation setting.
    Section 4.2 justifies the free-form answer design, but this makes all accuracy numbers dependent on the lenient rubric matching rules used by human and LLM judges.
  • domain assumption The human judges' rubric is consistent and applicable by LLM judges.
    Sections 5.1 and 5.2 share the same rubric between humans and LLMs; the ASD alignment result depends on this rubric being unambiguous and interpreted identically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data." pith.science (2026). https://pith.science/paper/DCEKGVSH

@misc{pith2026250718442,
  author       = {Pith},
  title        = {Pith review of: AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DCEKGVSH}},
  note         = {Machine review of arXiv:2507.18442}
}
read the original abstract

The cognitive and reasoning abilities of large language models (LLMs) have enabled remarkable progress in natural language processing. However, their performance in interpreting structured data, especially in tabular formats, remains limited. Although benchmarks for English tabular data are widely available, Arabic is still underrepresented because of the limited availability of public resources and its unique language features. To address this gap, we present AraTable, a novel and comprehensive benchmark designed to evaluate the reasoning and understanding capabilities of LLMs when applied to Arabic tabular data. AraTable consists of various evaluation tasks, such as direct question answering, fact verification, and complex reasoning, involving a wide range of Arabic tabular sources. Our methodology follows a hybrid pipeline, where initial content is generated by LLMs and subsequently filtered and verified by human experts to ensure high dataset quality. Initial analyses using AraTable show that, while LLMs perform adequately on simpler tabular tasks such as direct question answering, they continue to face significant cognitive challenges when tasks require deeper reasoning and fact verification. This indicates that there are substantial opportunities for future work to improve performance on complex tabular reasoning tasks. We also propose a fully automated evaluation framework that uses a self-deliberation mechanism and achieves performance nearly identical to that of human judges. This research provides a valuable, publicly available resource and evaluation framework that can help accelerate the development of foundational models for processing and analysing Arabic structured data.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 35 canonical work pages

  1. [1]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Li, X., Feng, X., Hu, S., Wu, M., Zhang, D., Zhang, J., Huang, K.: Dtllm-vlt: Diverse text generation for visual language tracking based on llm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7283–7292 (2024)

    work page 2024

  2. [2]

    SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation

    Liu, X., Sun, T., Xu, T., Wu, F., Wang, C., Wang, X., Gao, J.: Shield: Evaluation and defense strategies for copyright compliance in llm text generation. arXiv preprint arXiv:2406.12975 (2024)

    work page Pith review arXiv 2024

  3. [3]

    arXiv preprint arXiv:2402.04636 (2024)

    Koshkin, R., Sudoh, K., Nakamura, S.: Transllama: Llm-based simultaneous translation system. arXiv preprint arXiv:2402.04636 (2024)

    arXiv 2024

  4. [4]

    In: Findings of the Association for Computational Linguistics ACL 2024, pp

    Wang, L., Du, Z., Jiao, W., Lyu, C., Pang, J., Cui, L., Song, K., Wong, D., Shi, S., Tu, Z.: Benchmarking and improving long-text translation with large language models. In: Findings of the Association for Computational Linguistics ACL 2024, pp. 7175–7187 (2024)

    work page 2024

  5. [5]

    Allemang, D., Sequeda, J.: Increasing the llm accuracy for question answering: Ontologies to the rescue! arXiv preprint arXiv:2405.11706 (2024)

    arXiv 2024

  6. [6]

    In: Proceedings of the 1st ACM Workshop on AI-Powered Q&A Systems for Multimedia, pp

    Bui, T., Tran, O., Nguyen, P., Ho, B., Nguyen, L., Bui, T., Quan, T.: Cross-data knowledge graph construction for llm-enabled educational question-answering system: a case study at hcmut. In: Proceedings of the 1st ACM Workshop on AI-Powered Q&A Systems for Multimedia, pp. 36–43 (2024)

    work page 2024

  7. [7]

    Electronics 13(19), 3806 (2024)

    O’Brien Quinn, H., Sedky, M., Francis, J., Streeton, M.: Literature review of explainable tabular data analysis. Electronics 13(19), 3806 (2024)

    work page 2024

  8. [8]

    arXivabs/2412.12386 (2024)

    Nguyen, G., Brugere, I., Sharma, S., Kariyappa, S., Nguyen, A.T., Lecue, F.: Interpretable llm-based table question answering. arXivabs/2412.12386 (2024)

    arXiv 2024

Show all 62 references
  1. [9]

    In: Find- ings of the Association for Computational Linguistics: EMNLP 2024 (2024)

    Zhang, Z., Lin, X.V., Agrawal, R., Riedewald, M., Zhang, J.: Normtab: Improv- ing symbolic reasoning in llms through tabular data normalization. In: Find- ings of the Association for Computational Linguistics: EMNLP 2024 (2024). https://api.semanticscholar.org/CorpusID:272645535

    2024

  2. [10]

    arXiv abs/2403.19318 (2025)

    Wang, Y., Zhang, S., Lu, Y., Wang, J., Wang, S., Zhang, X., Li, H.: Tablellm: Enabling tabular data manipulation by llms in real office usage scenarios. arXiv abs/2403.19318 (2025)

    2025 arXiv

  3. [11]

    arXiv abs/2408.09174 (2024) 24

    Wu, X., Liang, D., Li, Z.: Tablebench: A comprehensive and complex benchmark for table question answering. arXiv abs/2408.09174 (2024) 24

    2024 arXiv

  4. [12]

    In: Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp

    Grijalba, J.O., Ure˜ na-L´ opez, L.A., Camacho-Collados, J., C´ amara, E.M.: Ques- tion answering over tabular data with databench: A large-scale empirical evalua- tion of llms. In: Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language...

    2024

  5. [13]

    arXiv preprint arXiv:2506.05587 (2025)

    Xing, J., He, Y., Zhou, M., Dong, H., Han, S., Chen, L., Zhang, D., Chaudhuri, S., Jagadish, H.: Mmtu: A massive multi-task table understanding and reasoning benchmark. arXiv preprint arXiv:2506.05587 (2025)

    2025

  6. [14]

    arXiv preprint arXiv:2407.21783 (2024)

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  7. [15]

    Hugging Face, https: //huggingface.co/mistralai/Mistral-Large (2024)

    AI, M.: Mistral-Large (Version 1.0) [Large language model]. Hugging Face, https: //huggingface.co/mistralai/Mistral-Large (2024)

    2024

  8. [16]

    arXiv preprint arXiv:2412.19437 (2024)

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  9. [17]

    arXiv preprint arXiv:2308.16149 (2023)

    Sengupta, N., Sahu, S.K., Jia, B., Katipomu, S., Li, H., Koto, F., Marshall, W., Gosal, G., Liu, C., Chen, Z., et al.: Jais and jais-chat: Arabic-centric founda- tion and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149 (2023)

    2023 arXiv

  10. [18]

    arXiv preprint arXiv:2505.08004 (2025)

    Rhel, H., Roussinov, D.: Large language models and arabic content: A review. arXiv preprint arXiv:2505.08004 (2025)

    2025 arXiv

  11. [19]

    ACL Anthology (2025)

    Mousi, S., al.: Aradice: Benchmarks for dialectal and cultural capabilities in llms. ACL Anthology (2025)

    2025

  12. [20]

    Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC) (2020)

    Sajjad, H., al.: Arabench: A multi-task benchmark for arabic natural lan- guage processing. Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC) (2020)

    2020

  13. [21]

    In: Habash, N., Bouamor, H., Hajj, H., Magdy, W., Zaghouani, W., Bougares, F., Tomeh, N., Abu Farha, I., Touileb, S

    Seelawi, H., Tuffaha, I., Gzawi, M., Farhan, W., Talafha, B., Badawi, R., Sober, Z., Al-Dweik, O., Freihat, A.A., Al-Natsheh, H.: ALUE: Arabic lan- guage understanding evaluation. In: Habash, N., Bouamor, H., Hajj, H., Magdy, W., Zaghouani, W., Bougares, F., Tomeh, N., Abu Far...

  14. [22]

    arXiv preprint arXiv:2109.00845 (2021) 25

    Abdul-Mageed, M., Elfardy, H., Diab, M.: Arlue: An arabic representation learning benchmark for universal evaluation. arXiv preprint arXiv:2109.00845 (2021) 25

    2021 arXiv

  15. [23]

    arXiv preprint arXiv:2506.08768 (2025)

    Hasanaath, A., Alansari, A., Ashraf, A., Salmane, C., Luqman, H., Ezzini, S.: Arareasoner: Evaluating reasoning-based llms for arabic nlp. arXiv preprint arXiv:2506.08768 (2025)

    2025

  16. [24]

    In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp

    Abdallah, A., Kasem, M., Abdalla, M., Mahmoud, M., Elkasaby, M., Elbendary, Y., Jatowt, A.: Arabicaqa: A comprehensive dataset for arabic question answer- ing. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, ...

    2024

  17. [25]

    arXiv preprint arXiv:2503.07833 (2025)

    Abdaljalil, S., Kurban, H., Serpedin, E.: Halluverse25: Fine-grained multilin- gual benchmark dataset for llm hallucinations. arXiv preprint arXiv:2503.07833 (2025)

    2025 arXiv

  18. [26]

    In: Proceedings of the 31st International Conference on Computational Linguistics, pp

    Alghamdi, E.A., Masoud, R., Alnuhait, D., Alomairi, A.Y., Ashraf, A., Zaytoon, M.: Aratrust: An evaluation of trustworthiness for llms in arabic. In: Proceedings of the 31st International Conference on Computational Linguistics, pp. 8664–8679 (2025)

    2025

  19. [27]

    In: Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp

    Sui, Y., Zhou, M., Zhou, M., Han, S., Zhang, D.: Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In: Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 645–654 (2024)

    2024

  20. [28]

    In: Duh, K., Gomez, H., Bethard, S

    Liu, T., Wang, F., Chen, M.: Rethinking tabular data understanding with large language models. In: Duh, K., Gomez, H., Bethard, S. (eds.) Proceed- ings of the 2024 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Techn...

    2024 doi

  21. [29]

    In: Findings of the Association for Computational Linguistics: ACL 2023, pp

    Ziqi, J., Lu, W.: Tab-cot: Zero-shot tabular chain of thought. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 10259–10277 (2023)

    2023

  22. [30]

    arXiv preprint arXiv:2401.04398 (2024)

    Wang, Z., Zhang, H., Li, C.-L., Eisenschlos, J.M., Perot, V., Wang, Z., Miculicich, L., Fujii, Y., Shang, J., Lee, C.-Y., et al.: Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv preprint arXiv:2401.04398 (2024)

    2024 arXiv

  23. [31]

    In: Findings of the Association for Computational Linguistics ACL 2024, pp

    Deng, N., Sun, Z., He, R., Sikka, A., Chen, Y., Ma, L., Zhang, Y., Mihalcea, R.: Tables as texts or images: Evaluating the table reasoning ability of llms and mllms. In: Findings of the Association for Computational Linguistics ACL 2024, pp. 407–426 (2024)

    2024

  24. [32]

    In: Vlachos, A., Augenstein, I

    Chen, W.: Large language models are few(1)-shot table reasoners. In: Vlachos, A., Augenstein, I. (eds.) Findings of the Association for Computational Lin- guistics: EACL 2023, pp. 1120–1130. Association for Computational Linguistics, Dubrovnik, Croatia (2023). https://doi.org/...

    2023 doi

  25. [33]

    In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024), pp

    Grijalba, J.O., Lopez, L.A.U., Mart ´ ınez-C´ amara, E., Camacho-Collados, J.: Question answering over tabular data with databench: A large-scale empirical evaluation of llms. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Res...

    2024

  26. [34]

    arXiv preprint arXiv:2406.12719 (2024)

    Bhandari, K.R., Xing, S., Dan, S., Gao, J.: On the robustness of language models for tabular question answering. arXiv preprint arXiv:2406.12719 (2024)

    2024 arXiv

  27. [35]

    arXiv preprint arXiv:2412.05579 (2024)

    Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., Liu, Y.: Llms-as- judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579 (2024)

    2024 arXiv

  28. [36]

    arXiv preprint arXiv:2305.13711 (2023)

    Lin, Y.-T., Chen, Y.-N.: Llm-eval: Unified multi-dimensional automatic evalua- tion for open-domain conversations with large language models. arXiv preprint arXiv:2305.13711 (2023)

    2023 arXiv

  29. [37]

    arXiv preprint arXiv:2303.16634 (2023)

    Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C.: G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634 (2023)

    2023 arXiv

  30. [38]

    arXiv preprint arXiv:2308.07201 (2023)

    Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., Liu, Z.: Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201 (2023)

    2023 arXiv

  31. [39]

    arXiv preprint arXiv:2401.15641 (2024)

    Chu, Z., Ai, Q., Tu, Y., Li, H., Liu, Y.: Pre: A peer review based large language model evaluator. arXiv preprint arXiv:2401.15641 (2024)

    2024 arXiv

  32. [40]

    arXiv e-prints, 2310 (2023)

    Li, Q., Cui, L., Kong, L., Bi, W.: Collaborative evaluation: Exploring the synergy of large language models and humans for open-ended generation evaluation. arXiv e-prints, 2310 (2023)

    2023

  33. [41]

    In: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp

    Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., Ma, X.: Towards human-ai deliberation: Design and evaluation of llm-empowered deliberative ai for ai-assisted decision-making. In: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1–23 (2025)

    2025

  34. [42]

    In: Proceedings of the 31st International Conference on Computational Linguistics, pp

    Lee, N., Hong, J., Thorne, J.: Evaluating the consistency of llm evaluators. In: Proceedings of the 31st International Conference on Computational Linguistics, pp. 10650–10659 (2025)

    2025

  35. [43]

    Advances in Neural Information Processing Systems 37, 68772– 68802 (2024)

    Panickssery, A., Bowman, S., Feng, S.: Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems 37, 68772– 68802 (2024)

    2024

  36. [44]

    In: Proceedings of the 18th ACM Conference on Recommender Systems, pp

    Zhang, X., Li, Y., Wang, J., Sun, B., Ma, W., Sun, P., Zhang, M.: Large language 27 models as evaluators for recommendation explanations. In: Proceedings of the 18th ACM Conference on Recommender Systems, pp. 33–42 (2024)

    2024

  37. [45]

    Pasupat, P., Liang, P.: Compositional semantic parsing on semi-structured tables. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. ...

    2015

  38. [46]

    Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F., Chua, T.- S.: Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1...

    2021

  39. [47]

    arXiv preprint arXiv:1909.02164 (2019)

    Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., Wang, W.Y.: Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164 (2019)

    2019 arXiv

  40. [48]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, vol

    Wu, X., Yang, J., Chai, L., Zhang, G., Liu, J., Du, X., Liang, D., Shu, D., Cheng, X., Sun, T., et al.: Tablebench: A comprehensive and complex benchmark for table question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 25497–25506 (2025)

    2025

  41. [49]

    In: Findings of the Association for Computational Linguistics: ACL 2023, pp

    Kweon, S., Kwon, Y., Cho, S., Jo, Y., Choi, E.: Open-wikitable: Dataset for open domain question answering with complex reasoning over table. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 8285–8297 (2023)

    2023

  42. [50]

    arXiv preprint arXiv:2506.03949 (2025)

    Zhu, J., Wang, J., Yu, B., Wu, X., Li, J., Wang, L., Xu, N.: Tableeval: A real- world benchmark for complex, multilingual, and multi-structured table question answering. arXiv preprint arXiv:2506.03949 (2025)

    2025

  43. [51]

    arXiv preprint arXiv:2406.10922 (2024)

    Berkovitch, Y., Glickman, O., Somech, A., Wolfson, T.: Generating tables from the parametric knowledge of language models. arXiv preprint arXiv:2406.10922 (2024)

    2024 arXiv

  44. [52]

    Elnagar, A., Khalifa, Y.S., Einea, A.: Hotel arabic-reviews dataset construction for sentiment analysis applications 740, 35–52 (2018) https://doi.org/10.1007/ 978-3-319-67056-0 3

    2018

  45. [53]

    arXiv preprint arXiv:2402.17944 (2024)

    Fang, X., Xu, W., Tan, F.A., Zhang, J., Hu, Z., Qi, Y., Nickleach, S., Socol- insky, D., Sengamedu, S., Faloutsos, C.: Large language models (llms) on tabular data: Prediction, generation, and understanding–a survey. arXiv preprint arXiv:2402.17944 (2024)

    2024 arXiv

  46. [54]

    Nguyen, N.-H., Sim, T., Dao, H., Joty, S., Kawaguchi, K., Chen, N., Kan, M.- Y., et al.: Llms are biased towards output formats! systematically evaluating and mitigating output format bias of llms. In: Proceedings of the 2025 Conference 28 of the Nations of the Americas Chapte...

    2025

  47. [55]

    In: Dernoncourt, F., Preot ¸iuc-Pietro, D., Shimorina, A

    Tam, Z.R., Wu, C.-K., Tsai, Y.-L., Lin, C.-Y., Lee, H.-y., Chen, Y.-N.: Let me speak freely? a study on the impact of format restrictions on large lan- guage model performance. In: Dernoncourt, F., Preot ¸iuc-Pietro, D., Shimorina, A. (eds.) Proceedings of the 2024 Conference ...

    2024 doi

  48. [56]

    arXiv preprint arXiv:2402.18667 (2024)

    Xia, C., Xing, C., Du, J., Yang, X., Feng, Y., Xu, R., Yin, W., Xiong, C.: Fofo: A benchmark to evaluate llms’ format-following capability. arXiv preprint arXiv:2402.18667 (2024)

    2024 arXiv

  49. [57]

    arXiv preprint arXiv:2202.01875 (2022)

    Lakkaraju, H., Slack, D., Chen, Y., Tan, C., Singh, S.: Rethinking explainability as a dialogue: A practitioner’s perspective. arXiv preprint arXiv:2202.01875 (2022)

    2022 arXiv

  50. [58]

    https://qwenlm.github

    Qwen: Qwen: A Large Language Model by Tongyi Lab. https://qwenlm.github. io/. Accessed: 2023-10-01 (2023)

    2023

  51. [59]

    Large language model (2024)

    OpenAI: GPT-4o (Version gpt-4o-latest). Large language model (2024). https: //openai.com/index/hello-gpt-4o

    2024

  52. [60]

    arXiv preprint arXiv:2409.14051 (2024)

    Liu, T., Wang, X., Huang, W., Xu, W., Zeng, Y., Jiang, L., Yang, H., Li, J.: Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion. arXiv preprint arXiv:2409.14051 (2024)

    2024

  53. [61]

    In: Forty-first International Conference on Machine Learning (2023) 29

    Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., Mordatch, I.: Improving factual- ity and reasoning in language models through multiagent debate. In: Forty-first International Conference on Machine Learning (2023) 29

    2023

  54. [184]

    https://aclanthology.org/2021.wanlp-1.18/

    Association for Computational Linguistics, Kyiv, Ukraine (Virtual) (2021). https://aclanthology.org/2021.wanlp-1.18/

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.