Pith. sign in

REVIEW 5 major objections 6 minor 28 references

Benchmarking Table Comprehension In The Wild

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that, in real table-rich financial reports, large language models can locate facts with reasonable accuracy but consistently falter on multi-step calculations and analytical reasoning, and introduces a benchmark…

desk verdict A creative benchmark construction undermined by an evaluation loop where the same model writes the gold answers, judges the outputs, and tops the ranking. read the letter →

arxiv 2412.09884 v1 pith:LBBR6JLX submitted 2024-12-13 cs.CL

Yikang Pan , Yi Zhu , Rand Xie , Yizhi Liu This is my paper · ORCID
classification cs.CL
keywords tablequestionansweringLLMevaluationfinancialreports10-Kfilingslong-contextmodelscomprehensionbenchmarknumericalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces TableQuest, a benchmark for testing whether large language models can read tables the way a human analyst does: not one isolated skill but extraction, calculation, and analytical reasoning together, inside the natural context of real financial filings. The authors compiled question–answer pairs from S&P 500 companies' 10-K reports, organised into easy, medium, and hard tiers, and probed seven state-of-the-art models. Their central finding is that the models do reasonably well at locating facts in a table but often fail at multi-step calculations and at synthesising what the numbers mean. If the benchmark is sound, it gives a more realistic measure of table comprehension than earlier single-skill benchmarks, and it suggests where future work on tabular reasoning should concentrate.

What carries the argument

The central object is TableQuest itself: question–answer pairs built from SEC EDGAR 10-K HTML tables, generated by GPT-4-Turbo through a progressive multi-round chat that moves from cell extraction (easy) to calculation (medium) to analytical synthesis (hard), then filtered by a hybrid human–machine pipeline that includes finance-expert guidelines and recall-based self-validation. The benchmark carries the argument because it places every question inside the document context where the skill must be exercised, rather than in an isolated table; the three difficulty levels are what let the paper separate fact-location from multi-step reasoning.

What would settle it

Take the 240 evaluation questions and have a panel of financial analysts with the table context produce their own gold answers and rate each model's response, then compare their ratings to the GPT-4-Turbo judge's ELO scores; a substantial mismatch, or a case where the judge prefers a response that contradicts the stated table numbers, would falsify the claim that the ranking measures table comprehension in the wild.

Watch

Extended reading notes

Core claim

On TableQuest, a dataset of 240 questions drawn from authentic 10-K reports, current LLMs display a split profile: they can extract a cell value or retrieve a fact with reasonable accuracy, but their performance drops when a question demands chained numerical operations or an analytical conclusion. The paper reports ELO scores with GPT-4o as a fixed baseline and GPT-4-Turbo as judge, and the resulting ranking places GPT-4-Turbo and Claude-3-5-Sonnet clearly above the open-source models, with the gap widest on hard questions. The authors interpret the pattern as evidence that holistic table comprehension, where a reader must combine retrieval, arithmetic, and domain awareness in one pass, remains a largely unsolved capability for current models.

Load-bearing premise

The load-bearing premise is that the synthetic question–answer pairs generated by GPT-4-Turbo are factually correct and that GPT-4-Turbo's judgments of model responses are fair; if either fails, the ranking and the claim about stumbling on hard questions would rest on unstable ground.

Editorial extensions

If this is right

  • Proprietary models such as GPT-4-Turbo and Claude-3-5-Sonnet lead the ELO ranking, and the gap over open-source models is largest on hard analytical questions.
  • Models such as Gemini-1.5-pro and Meta-Llama-3.1-70B score well on easy and medium questions but lose ground on hard questions, so overall ELO hides weaknesses that the difficulty split exposes.
  • Retrieving a fact from a table is not the bottleneck; multi-step calculations and query-focused analytical summaries are where current models fail.
  • Truncating inputs beyond the context limit affected fewer than 2% of samples, so the hard-question gap cannot be blamed on context-length limits.
  • The self-validation experiment, where the synthesis pipeline was re-run on academic papers and checked by recall against evidence sentences, supports the transferability of the data-generation procedure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same model family (GPT-4-Turbo) generated the gold answers and judged the responses, the absolute ELO ordering may partly reflect a model's stylistic affinity to its own outputs; a neutral human re-judging of a sample would test this.
  • The benchmark's design could be pushed further by forcing models to combine several tables or to challenge an incorrect premise, which would test whether the hard-question deficit is about arithmetic or about cross-table integration.
  • The reported ELO metric is relative to a GPT-4o baseline with a base score of 1000; swapping the baseline model would change the numbers but, if the paper is right, should preserve the qualitative ordering on hard tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces TableQuest, a benchmark for evaluating holistic table comprehension in real-world financial reports (S&P 500 10-K filings). The benchmark consists of 240 QA pairs at three difficulty levels (extraction, calculation, analytics), synthesized by GPT-4-Turbo with human-in-the-loop filtering. The authors evaluate seven text-based LLMs and two multimodal variants, reporting ELO ratings computed by a GPT-4-Turbo judge and, in the appendix, accuracy numbers. The central empirical claim is that current LLMs locate facts with reasonable accuracy but often falter on sophisticated reasoning or multi-step calculations. The paper also includes a qualitative failure-mode analysis and a self-supervised validation study on academic papers.

Significance. If the evaluation were trustworthy, TableQuest would fill a real gap: existing table QA benchmarks largely use isolated tables without surrounding context, whereas this benchmark uses full table-text documents. The three-level skill taxonomy and the use of pairwise ELO judging for open-ended analytics are interesting methodological ideas. The paper also ships a qualitative case study and an attempted self-supervised validation of the synthesis pipeline. However, the central empirical results (model ranking and the hard-vs-easy difficulty gradient) rest on a self-evaluation loop in which the same model generates the gold answers, serves as the judge, and tops the leaderboard. The provided validation does not break this loop, and the accuracy/ELO results are internally inconsistent. As a benchmark paper, it also fails to provide the promised public data or code. These issues collectively undermine the paper's main claims as presented.

major comments (5)
  1. [§2.1, §3, Appendix A, Table 2] The evaluation is circular: GPT-4-Turbo synthesizes the questions and reference answers (Section 2.1, Appendix A), serves as the judging model that compares model predictions against those reference answers (Section 3, Appendix A), and is the top-ranked model in Table 2. The judging prompt explicitly asks for 'semantic consistency with the gold answer' and even prefers 'verbatim' matches, creating an incentive for the judge to favor outputs resembling its own generated gold answers. Appendix B's human annotation covered only 50 questions per difficulty and was used to refine prompts, not to independently label the final 240-question evaluation subset. Appendix F validates synthesis on 2,100 academic-paper questions using token recall, but does not test judge impartiality or the correctness of the financial gold answers. The authors must provide an independent human evaluation on a sample of the final eval set, or a bias-controlled study with a blind judge, before the ranking and difficulty-gap claims can be accepted.
  2. [Table 3] The reported ELO ratings and accuracy percentages are inconsistent in ways that are not explained. For example, gpt-4o-vision has the highest overall accuracy (70.0%) but a lower overall ELO (892.46) than gpt-4-turbo (ELO 1164.35, accuracy 65.625%). Similarly, Meta-Llama-3.1-70B-Instruct has identical accuracy (48.75%) across overall, easy, and medium columns but its ELO varies substantially across difficulties (880.19, 1008.80, 703.98). If ELO and accuracy are meant to measure related constructs, these discrepancies require an explanation; if they measure different things, the relationship should be stated explicitly. The paper should report per-difficulty accuracy and the exact ELO computation details, including the number of pairwise comparisons per model pair and the handling of ties.
  3. [§3, Table 3, Results] No error bars, confidence intervals, or significance tests are provided anywhere in the paper. With only 80 questions per difficulty level, differences such as the 1.46-point ELO gap between Qwen2-72B-Instruct (867.81) and Meta-Llama-3.1-70B-Instruct (869.27) are almost certainly within noise. The authors should report bootstrap confidence intervals or another uncertainty quantification for both ELO and accuracy, and avoid drawing strong conclusions from small differences.
  4. [Appendix F and Appendix B] The self-supervised validation in Appendix F does not validate the central claims. It achieves an average token-recall of only 0.736 on a different domain (academic papers), which is a weak semantic metric and does not establish that the financial gold answers are correct. Moreover, the validation does not assess the impartiality of the GPT-4-Turbo judge. Appendix B's human annotation is limited to 50 questions per difficulty and is explicitly described as used for prompt refinement, not as an independent label of the final 240-question eval set. These gaps leave the correctness of the gold answers and the judge's behavior unverified.
  5. [Abstract, §5, Availability] The abstract and conclusion state that 'the evaluation data, judging procedure and results of this study' are made publicly available, but the paper provides no link, repository, or contact for obtaining them. For a benchmark paper, the data and code are the main artifact; without them, the benchmark cannot be used or independently checked. The authors must provide a working URL or a clear statement of how to access the data and evaluation scripts.
minor comments (6)
  1. [§4] There is a grammatical error in the sentence 'We observe that among open-source model, However, Meta-Llama-3.1-70B-Instruct...' — the word 'However' is misplaced and the sentence is incomplete.
  2. [Table 3] The column headers in Table 3 are ambiguous: 'ELO Rating Accuracy (%) Overall Hard Overall Easy Medium' does not clearly map which columns are ELO and which are accuracy. Please label them explicitly, e.g., 'ELO (Overall)', 'ELO (Hard)', 'Accuracy (Overall)', 'Accuracy (Easy)', 'Accuracy (Medium)'.
  3. [References] Reference [20] (MultiTabQA) is missing publication venue and year details, and appears incomplete.
  4. [Appendix A] The temperature and other decoding parameters for the GPT-4-Turbo judge are not specified; the authors only state temperature 0.05 for the evaluated models. This should be reported for reproducibility.
  5. [Appendix B] No inter-annotator agreement or quantitative acceptance/rejection rates are reported for the human annotation step, despite the claim of maintaining 'a high standard of accuracy.' Provide numbers such as Cohen's kappa or the fraction of questions edited/rejected.
  6. [Figure 1] The figure caption mentions a sample image rendered from HTML, but the figure itself is not described in the text; please refer to it in Section 1 or 2 and ensure it is legible in the final version.

Circularity Check

1 steps flagged · score 6.0 of 10

The benchmark's central ranking and hard-task claims rest on GPT-4-Turbo judging other models against gold answers GPT-4-Turbo itself generated, with the judge instructed to prefer verbatim overlap with that gold, making the top result partially self-confirming.

  1. self definitional [Section 3 (Evaluation Framework); Appendix A (judging prompt); Appendix F (synthesis model identification); Table 2]
    "For analytics tasks, we compare each model’s response to a baseline (in our case, gpt-4o using text input, with a 1000 base score), ask the judging model (gpt-4-turbo) to reason about which model is more accurate, logical, and clear in its output. ... Accuracy: The assistant’s answer is semantically consistent with the gold answer, preferably with the reference answer verbatim."

    The gold answers that define correctness were produced by GPT-4-Turbo (Appendix F: 'our synthesis model’s (GPT-4-turbo)'), and the judge is the same gpt-4-turbo. The judging prompt operationalizes 'accurate' as semantic consistency with that same model's gold, preferably verbatim. GPT-4-Turbo then appears as the top-ranked model (overall ELO 1164.35, Table 2). The hard-task ELOs supporting the abstract's 'often falter' claim are likewise computed by this self-referential judge. The Appendix F validation uses token recall on academic-paper questions and does not test judge impartiality or independently relabel the 240 financial evaluation questions, so the ranking and the difficulty gradient are partially defined by GPT-4-Turbo's agreement with itself.

full rationale

The paper's main empirical claim - that LLMs locate facts reasonably but falter on complex reasoning in table-rich reports - is an evaluation result rather than a mathematical derivation. The evaluation loop is the load-bearing step: Section 2.1 synthesizes question-answer pairs with GPT-4-Turbo, Section 3 appoints gpt-4-turbo as the judge, Appendix A instructs the judge to score answers by semantic consistency with the reference answer 'preferably ... verbatim', and Table 2 reports GPT-4-Turbo as the top model. This makes the performance measure partly self-referential: a model is rewarded for reproducing the phrasing and reasoning style of the model that wrote the gold answers. The human annotation in Appendix B covered only 50 questions per difficulty and was used to refine the generation prompts, not to independently label the final 240-question evaluation subset, so it does not break the loop. Appendix F's self-supervised validation reports 0.736 average token recall on 2,100 academic-paper questions and explicitly keeps all synthesis settings identical; it checks grounding of generated answers, not the impartiality of the same model as judge, and it is in a different domain. Because extraction and calculation sub-tasks have objective accuracy numbers in Appendix D and the human refinement provides some external input, the circularity is substantial but not total. This is not a self-citation or ansatz-smuggling case; it is a definitional evaluation loop, so the score is 6 rather than higher.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on the correctness of synthetic QA pairs, the impartiality of the LLM judge, and the adequacy of HTML table serialization. The paper provides only weak self-validation for these assumptions.

free parameters (1)
  • ELO base rating = 1000
    Arbitrary anchor used to initialize ratings; affects absolute ELO but not relative ranking.
assumptions (4)
  • domain assumption GPT-4-Turbo generated question-answer pairs are factually correct and appropriately difficult after human filtering.
    Human annotation covered only 50 questions per difficulty, and Appendix F's recall validation is on academic papers, not financial QA.
  • ad hoc to paper GPT-4-Turbo is an impartial judge of analytical answers across models.
    The same model family serves as synthesizer, judge, and top-scoring model; no calibration against human judgments or other judges is reported.
  • domain assumption HTML serialization preserves enough table structure for evaluating table comprehension.
    Used for all inputs; no comparison to alternative serializations in the experiments.
  • standard math ELO rating system and accuracy metrics as implemented are valid for the 240-question eval set.
    ELO is a standard method, but no confidence intervals are reported for small samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Table Comprehension In The Wild." pith.science (2026). https://pith.science/paper/LBBR6JLX

@misc{pith2026241209884,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Table Comprehension In The Wild},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LBBR6JLX}},
  note         = {Machine review of arXiv:2412.09884}
}
read the original abstract

Large Language Models (LLMs), while being increasingly dominant on a myriad of knowledge-intensive activities, have only had limited success understanding lengthy table-text mixtures, such as academic papers and financial reports. Recent advances of long-context LLMs have opened up new possibilities for this field. Nonetheless, we identify two roadblocks: (1) Prior benchmarks of table question answering (TableQA) have focused on isolated tables without context, making it hard to evaluate models in real-world scenarios. (2) Prior benchmarks have focused on some narrow skill sets of table comprehension such as table recognition, data manipulation/calculation, table summarization etc., while a skilled human employs those skills collectively. In this work, we introduce TableQuest, a new benchmark designed to evaluate the holistic table comprehension capabilities of LLMs in the natural table-rich context of financial reports. We employ a rigorous data processing and filtering procedure to ensure that the question-answer pairs are logical, reasonable, and diverse. We experiment with 7 state-of-the-art models, and find that despite reasonable accuracy in locating facts, they often falter when required to execute more sophisticated reasoning or multi-step calculations. We conclude with a qualitative study of the failure modes and discuss the challenges of constructing a challenging benchmark. We make the evaluation data, judging procedure and results of this study publicly available to facilitate research in this field.

Figures

Figures reproduced from arXiv: 2412.09884 by the authors.

Figure 1
Figure 1. A sample of our TableQuest benchmark showcasing the different aspects of table compre [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Excerpt from the company’s financial report [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Input lengths statistics. Length Distribution (%) Recall 0-5 26.7% 0.704 5-10 15.0% 0.788 10-15 19.3% 0.816 15-20 15.8% 0.782 20+ 23.2% 0.642 Avg. 100% 0.736 [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 14 canonical work pages

  1. [1]

    Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA, June

    Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA, June

    work page

  2. [2]

    FinanceBench: A New Benchmark for Financial Question Answering, November 2023

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. FinanceBench: A New Benchmark for Financial Question Answering, November 2023. arXiv:2311.11944 [cs, stat]

    arXiv 2023

  3. [3]

    FinQA: A Dataset of Numerical Reasoning over Financial Data

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A Dataset of Numerical Reasoning over Financial Data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3697–3711, Online and Punta Cana, ...

    work page 2021

  4. [4]

    Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context, December 2020

    Xinyi Zheng, Doug Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context, December 2020. arXiv:2005.00589 [cs]

    arXiv 2020

  5. [5]

    TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios, April 2024

    Xiaokang Zhang, Jing Zhang, Zeyao Ma, Yang Li, Bohan Zhang, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, Jifan Yu, Shu Zhao, Juanzi Li, and Jie Tang. TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios, April 2024. arXiv:2403.19318 [cs]

    arXiv 2024

  6. [6]

    Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das

    Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. ToTTo: A Controlled Table-To-Text Generation Dataset, October

    work page

  7. [7]

    FeTaQA: Free-form Table Question Answering

    Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kry´sci´nski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. FeTaQA: Free-form Table Question Answering. Transactions of the Association...

    work page 2022

  8. [8]

    Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs

    Naihao Deng, Zhenjie Sun, Ruiqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihalcea. Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024 , pages 407–426, Bangkok, Thailand and virtual ...

    work page 2024

Show all 28 references
  1. [9]

    FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains, November 2023

    Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains, November 2023

    2023

  2. [10]

    TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance. In Proceedings of the 59th Annual Meeting of the Association for Computat...

    2021

  3. [11]

    Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table, May 2023

    Sunjun Kweon, Yeonsu Kwon, Seonhee Cho, Yohan Jo, and Edward Choi. Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table, May 2023. arXiv:2305.07288 [cs]

    2023 arXiv

  4. [12]

    Multi-modal Retrieval of Tables and Texts Using Tri-encoder Models, October 2021

    Bogdan Kosti´c, Julian Risch, and Timo Möller. Multi-modal Retrieval of Tables and Texts Using Tri-encoder Models, October 2021. arXiv:2108.04049 [cs]

    2021 arXiv

  5. [13]

    Mixed- modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA, October 2022

    Junjie Huang, Wanjun Zhong, Qian Liu, Ming Gong, Daxin Jiang, and Nan Duan. Mixed- modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA, October 2022. arXiv:2210.05197 [cs]. 5

    2022 arXiv

  6. [14]

    Needle in a haystack

    Kamradt G. Needle in a haystack. https://github.com/gkamradt/LLMTest_ NeedleInAHaystack, 2024

    2024

  7. [15]

    InternLM2 Technical Report, March 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang,...

    2024 arXiv

  8. [16]

    Large Language Mod- els(LLMs) on Tabular Data: Prediction, Generation, and Understanding – A Survey, June 2024

    Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large Language Mod- els(LLMs) on Tabular Data: Prediction, Generation, and Understanding – A Survey, June 2024. arXiv:2402....

    2024 arXiv

  9. [17]

    Large Language Model for Table Processing: A Survey, February 2024

    Weizheng Lu, Jiaming Zhang, Jing Zhang, and Yueguo Chen. Large Language Model for Table Processing: A Survey, February 2024. arXiv:2402.05121 [cs]

    2024 arXiv

  10. [18]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, September 2023. arXiv:1910.10683 [cs, stat]

    2023 arXiv

  11. [19]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, June 2024

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, June 2024. arXiv:2406.17557 [cs]

    2024 arXiv

  12. [20]

    MultiTabQA: Generat- ing Tabular Answers for Multi-Table Question Answering

    Vaishali Pal, Evangelos Kanoulas, Andrew Yates, and Maarten de Rijke. MultiTabQA: Generat- ing Tabular Answers for Multi-Table Question Answering

  13. [21]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  14. [22]

    Bleu: a Method for Automatic Evaluation of Machine Translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a Method for Automatic Evaluation of Machine Translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , ...

    2002

  15. [23]

    ROUGE: A Package for Automatic Evaluation of Summaries

    Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. InText Summariza- tion Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics

    2004

  16. [24]

    METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare V oss, editors,Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures...

    2005

  17. [25]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating Text Generation with BERT, February 2020. arXiv:1904.09675 [cs]

    2020 arXiv

  18. [26]

    steps"), then provide a succinct answer (marked by

    Minzheng_Wang. MozerWang/Loong, August 2024. original-date: 2024-06-16T12:21:44Z. 6 A Prompts Below is the inference prompt which we use on the 7 models tested. [System] You are a helpful question-answering chatbot. [User] You are asked to answer questions based on provided ma...

    2024

  19. [2020]

    arXiv:2004.14373 [cs]

    2004 arXiv

  20. [2024]

    arXiv:2406.17419 [cs]

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.