Pith. sign in

REVIEW 4 major objections 5 minor 33 references

SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read 953-question benchmark tests joint table-and-text reasoning

desk verdict SciTaT targets a real gap in scientific QA, but the contradictory generation prompt and unmeasured gold-label quality mean the joint-reasoning claim is not yet established. read the letter →

arxiv 2412.11757 v1 pith:P5GFQ7JI submitted 2024-12-16 cs.CL

Xuanliang Zhang , Dingzirui Wang , Baoxin Wang , Longxu Dou , Xinyuan Lu , Keyan Xu , Dayong Wu , Qingfu Zhu
show 1 more author
Wanxiang Che
This is my paper · ORCID
classification cs.CL
keywords scientificquestionansweringtable-and-textreasoningQAbenchmarktaxonomylargelanguagemodelsdataanalysistabulationchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Scientific question answering stops at reading either text or tables; this paper argues that the real questions researchers ask cut across both and demand a wider spread of reasoning moves. It introduces SciTaT, 953 questions drawn from 871 arXiv papers, organised into 4 reasoning types and 13 subtypes that include look-up, numerical reasoning, data analysis, and tabulation. The paper's central claim is that this benchmark is harder and more realistic than existing scientific QA datasets because over a third of its questions require combining information from tables and text in the same answer. To show the benchmark is usable, the authors build CaR, a baseline that first computes numerical information with code and then reasons over it, reporting an average 12.9% improvement over direct, chain-of-thought, and program-of-thought prompting. If the benchmark is sound, the field gains a stress test for the kinds of analytical questions working scientists actually ask.

What carries the argument

The load-bearing object is the SciTaT benchmark itself, with its four-by-thirteen reasoning taxonomy and its enforced table-and-text contexts. The taxonomy is the mechanism that differentiates the benchmark: it was induced from 650 questions sampled from a researcher-facing QA platform plus reasoning types from earlier datasets, and each subtype has a prescribed question format and evaluation behavior. The other mechanism is the CaR baseline, which splits the task into a Calculator and a Reasoner: the Calculator is prompted to write and run code that returns a complete sentence of numerical information instead of a bare answer, and the Reasoner performs chain-of-thought reasoning over that information while explicitly reflecting on whether the information is correct. The combination is what produces the paper's claimed 12.9% average improvement, and the ablation shows removing either module or reversing their order degrades performance.

What would settle it

Take a stratified random sample of SciTaT instances and have independent domain-expert annotators answer each question blind to the published gold answer, then measure agreement and the proportion of questions that are actually answerable from their context; if a substantial share (say more than 10%) of gold answers are wrong or unanswerable, the benchmark's central validity claim collapses. A second check is to rerun CaR against chain-of-thought over multiple random seeds and check whether the 12.9% average improvement persists.

Watch

Extended reading notes

Core claim

The central claim is that realistic scientific question answering must force models to reason jointly over tables and text, and that this requirement can be measured with a benchmark whose reasoning types come from real researcher queries rather than dataset convenience. The paper develops SciTaT, a 953-question benchmark from 871 papers, with four reasoning types and 13 subtypes, including Data Analysis and Tabulation that earlier datasets mostly omit. Each instance pairs a question with a rationale and an answer source labelled Text, Table, or Table-and-Text, and more than one-third of the questions require both modalities. On this resource, gpt-4o and Llama3.1 models all fall short of 50% exact match and F1 under the best baseline, and the paper takes this as evidence that SciTaT measures something current systems cannot yet do reliably.

Load-bearing premise

The benchmark's validity rests on the assumption that gpt-4o-drafted questions and answers, after light human refinement, are accurate and unbiased enough to serve as gold labels, yet the paper reports no inter-annotator agreement, no post-refinement error counts, and no record of how much editing changed the drafts.

Editorial extensions

If this is right

  • Even the strongest tested pipeline, CaR with gpt-4o, stays below 50% exact string match on short answers and below 50% F1 on free-form answers, so SciTaT marks current LLM performance as insufficient for scientific analysis questions.
  • Data Analysis is the hardest reasoning type, and Tabulation produces high lexical overlap with the gold answer but near-zero exact match, suggesting tabulation needs a different evaluation metric than string equality.
  • CaR's margin over chain-of-thought is larger in the long-context whole-paper setting, which points to numerical pre-extraction as a way to reduce the burden of grounding in long scientific documents.
  • Free-form answers and short-form answers fail in different ways, including omissions, grounding errors, and redundancy, so progress on SciTaT likely requires separate treatments for the two answer regimes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run is a human re-annotation study: because the gold labels originate from gpt-4o, an independent blind re-answer of a random sample would quantify label noise and show how much of the 12.9% gap is real model behaviour versus annotation bias.
  • The reasoning taxonomy could be transplanted to other scientific domains, such as biomedicine or chemistry, where the same four reasoning families should recur; a testable prediction is that data-analysis and tabulation questions remain the hardest there as well.
  • Since the questions were generated to require both tables and text, SciTaT likely rewards models that explicitly fuse the two modalities; comparing CaR against a retrieval-augmented model in the long-context setting would isolate whether the gain comes from locating evidence or from computing numbers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SCITAT, a question-answering benchmark for scientific tables and text, containing 953 questions derived from 871 arXiv papers in AI/ML. The benchmark covers four reasoning types (Look Up, Numerical Reasoning, Data Analysis, Tabulation) with 13 subtypes, and includes two evaluation settings, long-context and short-context. The authors also propose a baseline method, CAR, which combines a Calculator module (that generates and executes code to extract numerical information) with a Reasoner module (that performs final reasoning), reporting an average improvement of 12.9% over direct QA, CoT, and PoT baselines. Dataset construction relies on gpt-4o-generated questions, rationales, and answers, followed by human refinement and two-round validation. The paper also provides error analysis and discusses challenges such as grounding, complex calculation, and domain knowledge.

Significance. If the validity of the gold labels and the joint table-text requirement are established, SCITAT would fill a real gap in scientific QA by including Data Analysis and Tabulation reasoning types that are rare in existing datasets, and by including a substantial fraction (331/953, about 35%) of questions whose answer sources are both tables and text. The proposed CAR baseline is a reasonable first attempt at combining program-aided computation with free-form reasoning, and the authors publicly release their code and data. The main results, including the 12.9% average improvement and the finding that gpt-4o still scores below 50% EM/F1, would be useful to the community. However, the current manuscript does not provide sufficient evidence for the benchmark's core properties, namely the joint-reasoning requirement and the correctness/unbiasedness of the gold labels, and the experimental claims lack statistical grounding.

major comments (4)
  1. [Section 2.2 and Appendix B.1 (Table 8)] The question-generation prompt in Appendix B.1 states that the question 'must be answerable based on the paragraph alone, and not answerable only based on the table,' which directly contradicts the requirement in Figure 3 that 'the question is best answered by referring to both the tables and the text simultaneously.' This contradiction is load-bearing because the 331 TaT questions in Table 3 are the basis for the claim that SCITAT requires joint reasoning over tables and text. If the printed prompt was used as-is, many generated questions may be answerable from the paragraph alone, and the annotator-selected answer source (Table 3) would not verify that both sources are actually needed. The authors must resolve this discrepancy and provide evidence that the 331 TaT questions truly require joint reasoning, for example by reporting how often human refinement changed the answer source, or by conducting a post-hoc verification on a sample that the paragraph alone is insufficient.
  2. [Sections 2.3 and 2.4] The paper reports no inter-annotator agreement, no post-refinement error rate, and no quantitative measure of how much human editing altered the gpt-4o-generated questions, rationales, and answers. Since the gold labels are the yardstick for all model evaluations and for the headline 12.9% improvement, the absence of such quality metrics leaves the validity of the benchmark unquantified. The authors should report IAA (e.g., Cohen's kappa on answer-source selection and on answer correctness), a sampled post-refinement error rate (e.g., on a held-out set independently re-annotated), and statistics on the fraction of instances whose question, rationale, or answer was changed during refinement.
  3. [Section 4.2, Table 5] All experimental results are reported as single runs with no variance or significance testing. The claim that CAR improves over baselines by an average of 12.9% could be within run-to-run noise given the inherent stochasticity of LLM prompting. The authors should provide multiple runs (with standard deviations) for at least the main comparisons, and ideally paired significance tests (e.g., bootstrap or approximate randomization) to support the conclusion that CAR's improvements are systematic.
  4. [Section 4.1.1 and Table 5] The evaluation protocol is ambiguous: the text says short-form answers are evaluated with EM and free-form answers with F1/BERTScore, but Table 5 reports a single set of EM, F1, and BERTScore per setting without stating the subset each metric is computed on. Figure 5's statement that F1 and BERTScore are 0 for Look Up (all short-form) and EM is 0 for Data Analysis (all free-form) suggests that metrics are computed over all questions, which would mean free-form answers automatically receive EM 0 and short-form answers automatically receive F1 0. The authors must clarify the exact aggregation rule, the denominators for each metric (number of short-form vs. free-form questions), and how the 12.9% average improvement is computed across these heterogeneous metrics.
minor comments (5)
  1. [Section 2.2] The description of how the 650 SparkRA questions were categorized into reasoning types is brief; please specify whether this categorization was performed by the authors or by annotators, and whether any agreement measure was used for the categorization.
  2. [Section 2.1] The paper does not report the number of papers initially collected, how many were removed by the filtering criteria (papers without tables, irrelevant paragraphs, etc.), and the final yield rate; including these statistics would improve the transparency of dataset construction.
  3. [Table 1 and Table 7] The comparison tables use check marks and crosses for reasoning types, but Table 7 does not include the same set of datasets as Table 1, and the meaning of 'R' (Rationale) in Table 7 is not defined in the caption; please make the comparison fully consistent and self-explanatory.
  4. [Appendix C.2] The annotation statistics report 10 annotators for the first round and 2 for validation, but the earlier description in Section 2.4.1 mentions a training requirement with an Exact Match threshold of 95%; please clarify how many annotators passed the training and whether the two-round validation was performed by the same or different annotators across instances.
  5. [Limitations] The limitations section lists language and single-turn scope, but does not mention the potential limitation of reliance on gpt-4o-generated questions or the lack of external validation of the gold labels; given that the benchmark's validity is central, this should be acknowledged.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SciTaT's construction and CAR's 12.9% gain rest on human-refined labels and fixed baselines, not on inputs defined in terms of outputs.

full rationale

The paper contains no formal derivation, so the circularity question reduces to whether any load-bearing result is forced by construction. The benchmark labels were not defined in terms of the reported model performances: questions and answers were initially generated by gpt-4o, but Section 2.3 describes human refinement and Section 2.4.2 describes two-round validation, and the main experiments evaluate both gpt-4o and Llama3.1 against fixed Direct QA, CoT, and PoT prompts. CAR's 12.9% average improvement is an empirical comparison, not a fitted parameter renamed as a prediction; neither the Calculator nor the Reasoner is trained on SciTaT, and no result is obtained by plugging the benchmark's own answers back into the method. The self-citations, including SparkRA as a source of real-user reasoning types and Hpropro or TableBench as related work, are data-source or precedent citations rather than load-bearing uniqueness or ansatz claims. One genuine validity concern is the Appendix B.1 question-generation prompt requiring questions to be "answerable based on the paragraph alone, and not answerable only based on the table," which conflicts with the main-text requirement that questions be best answered by referring to both tables and text simultaneously; this affects whether the TaT subset truly requires joint reasoning. However, that is a consistency and gold-label validity issue, not a circular reduction: the benchmark's construction does not presuppose the performance gain being reported, and CAR's improvement is not equivalent by construction to any input or label. The distribution of reasoning types is also a design choice because questions were generated to match the taxonomy, but the paper presents this as a construction description rather than as a prediction, so it does not meet the standard for circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities and fits no free parameters. Its claims rest on domain assumptions about the representativeness of SparkRA questions, the quality of LLM-generated and human-checked labels, and the sufficiency of the selected context.

assumptions (3)
  • domain assumption The reasoning types observed in 650 questions on SparkRA are representative of real researcher queries in AI/CL/ML.
    Section 2.2 states: 'We randomly select 650 questions and categorize their reasoning types... We hypothesize that the reasoning types observed in these questions are comparable to those found in real-world inquiries.'
  • domain assumption gpt-4o-generated questions, rationales, and answers, after human refinement, are accurate enough to serve as gold labels.
    Sections 2.2 and 2.3 rely on this assumption; no inter-annotator agreement or post-refinement error rates are reported.
  • domain assumption A single randomly selected paragraph and its mentioned tables provide sufficient context to answer each question in the short-context setting.
    Section 2.1: 'the context we provide when annotating the question is a paragraph that mentions tables and the tables mentioned.' Some questions may require multiple paragraphs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types." pith.science (2026). https://pith.science/paper/P5GFQ7JI

@misc{pith2026241211757,
  author       = {Pith},
  title        = {Pith review of: SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P5GFQ7JI}},
  note         = {Machine review of arXiv:2412.11757}
}
read the original abstract

Scientific question answering (SQA) is an important task aimed at answering questions based on papers. However, current SQA datasets have limited reasoning types and neglect the relevance between tables and text, creating a significant gap with real scenarios. To address these challenges, we propose a QA benchmark for scientific tables and text with diverse reasoning types (SciTaT). To cover more reasoning types, we summarize various reasoning types from real-world questions. To involve both tables and text, we require the questions to incorporate tables and text as much as possible. Based on SciTaT, we propose a strong baseline (CaR), which combines various reasoning methods to address different reasoning types and process tables and text at the same time. CaR brings average improvements of 12.9% over other baselines on SciTaT, validating its effectiveness. Error analysis reveals the challenges of SciTaT, such as complex numerical calculations and domain knowledge.

Figures

Figures reproduced from arXiv: 2412.11757 by the authors.

Figure 1
Figure 1. Illustrations of the reasoning types in SCITAT. The tables and text (left) show color-coded spans for question context. The questions (right) are examples of 4 reasoning types, with their rationales and answers. the types summarized from the real questions in SparkRA (Wu et al., 2024a) and previous works (Lu et al., 2023; Wu et al., 2024b). SCITAT not only requires the model to look up information and numerical reas… view at source ↗
Figure 2
Figure 2. The overview of our annotation process. The [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The requirements for generating questions. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The overview of CAR, which consists of two modules: (i) Calculator generates code to compute the numerical information required for solving the question. (ii) Reasoner continues the reasoning process based on the information provided by the Calculator to answer the que…
Figure 5
Figure 5. Figure 5: The average performance of CAR across three models on four reasoning types. soner and avoiding the need to search for answers directly within the whole paper. Answer Types CAR shows more significant im￾provements in short-form answers than free-form answers. For short-…
Figure 6
Figure 6. Figure 6: The distribution of error types of CAR on free-form answers and short-form answers. and reversing the order of the two modules. Specif￾ically, when only the Reasoner is retained, it is the same as the CoT baseline. When reversing, we first apply the Reasoner module and…
Figure 7
Figure 7. Figure 7: The user interface, showing the paper information and the paragraph. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The user interface, showing the tables [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The user interface, showing the annotation information. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: The user interface, showing the choice for the answer source, relevant tables, and if to remain. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: The case for the error type of "Miss". Question: How does the calculated cumulative number of bi-sentences by the end of a Bachelor's degree differ between Analysis 1 and Analysis 2 based on the interpretation of the data in Table \ref{tab1}? Predicted Answer: We cann…
Figure 12
Figure 12. Figure 12: The case for the error type of "Grounding". [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: The case for the error type of "Calculation". [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: The case for the error type of "Knowledge". [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: The case for the error type of "Redundancy". [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 7 canonical work pages

  1. [1]

    Microsoft Research AI4Science and Microsoft Azure Quantum. 2023. https://arxiv.org/abs/2311.07361 The impact of large language models on scientific discovery: a preliminary study using gpt-4 . Preprint, arXiv:2311.07361

    arXiv 2023

  2. [2]

    Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587--604

    2018

  3. [3]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. https://openreview.net/forum?id=YfZ4ZPt8zd Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks . Transactions on Machine Learning Research

    2023

  4. [4]

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.91 H ybrid QA : A dataset of multi-hop question answering over tabular and textual data . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026--1036, Online. Association for Computational...

    doi:10.18653/v1/2020.findings-emnlp.91 2020

  5. [5]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.300 F in QA : A dataset of numerical reasoning over financial data . In Proceedings of the 2021 Conference on Empirical Methods in Natural Lang...

    doi:10.18653/v1/2021.emnlp-main.300 2021

  6. [6]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. https://doi.org/10.18653/v1/2021.naacl-main.365 A dataset of information-seeking questions and answers anchored in research papers . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te...

    doi:10.18653/v1/2021.naacl-main.365 2021

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

    arXiv 2024

  8. [8]

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

Show all 33 references
  1. [9]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. https://doi.org/10.18653/v1/D19-1259 P ub M ed QA : A dataset for biomedical research question answering . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing an...

    2019 doi

  2. [10]

    Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-In Lee, and Moontae Lee. 2023. https://proceedings.mlr.press/v202/lee23n.html QASA : Advanced question answering on scientific articles . In Proceedings of the 40th International Conference on Machine Le...

    2023

  3. [11]

    Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.483 SCITAB : A challenging benchmark for compositional reasoning and claim verification on scientific tables . In Proceedings of the 2023 Conference on Empiri...

    2023 doi

  4. [12]

    Nafise Sadat Moosavi, Andreas R \"u ckl \'e , Dan Roth, and Iryna Gurevych. 2021. Scigen: a dataset for reasoning-aware text generation from scientific tables. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

    2021

  5. [13]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  6. [14]

    Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. https://doi.org/10.18653/v1/D18-1258 emr QA : A large corpus for question answering on electronic medical records . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,...

    2018 doi

  7. [15]

    Dimitris Pappas, Ion Androutsopoulos, and Haris Papageorgiou. 2018. https://aclanthology.org/L18-1439 B io R ead: A new dataset for biomedical reading comprehension . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC 2018) , Mi...

    2018

  8. [16]

    Dimitris Pappas, Petros Stavropoulos, Ion Androutsopoulos, and Ryan McDonald. 2020. https://doi.org/10.18653/v1/2020.bionlp-1.15 B io MRC : A dataset for biomedical machine reading comprehension . In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing,...

    2020 doi

  9. [17]

    Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. 2024. Spiqa: A dataset for multimodal question answering on scientific papers. NeurIPS

    2024

  10. [18]

    Qi Shi, Han Cui, Haofeng Wang, Qingfu Zhu, Wanxiang Che, and Ting Liu. 2024. https://doi.org/10.18653/v1/2024.acl-long.594 Exploring hybrid question answering via program-based prompting . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    2024 doi

  11. [19]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  12. [20]

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085

    2022 arXiv

  13. [21]

    George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexi...

    2015

  14. [22]

    Dingzirui Wang, Longxu Dou, and Wanxiang Che. 2022. A survey on table-and-text hybridqa: Concepts, methods, challenges and future directions. arXiv preprint arXiv:2212.13465

    2022 arXiv

  15. [23]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  16. [24]

    Dayong Wu, Jiaqi Li, Baoxin Wang, Honghong Zhao, Siyuan Xue, Yanjie Yang, Zhijun Chang, Rui Zhang, Li Qian, Bo Wang, Shijin Wang, Zhixiong Zhang, and Guoping Hu. 2024 a . https://aclanthology.org/2024.emnlp-demo.40 S park RA : A retrieval-augmented knowledge service system bas...

    2024

  17. [25]

    Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. 2024 b . Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXiv:2408.09174

    2024 arXiv

  18. [26]

    Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. https://openreview.net/forum?id=LC1QAqhePv Sciinstruct: a self-reflective instruction annotated dataset for training scientific language models . In The Thi...

    2024

  19. [27]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  20. [28]

    Yilun Zhao, Lyuhao Chen, Arman Cohan, and Chen Zhao. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.692 T a PERA : Enhancing faithfulness and interpretability in long-form table QA by content planning and execution-based reasoning . In Proceedings of the 62nd Annual Meetin...

    2024 doi

  21. [29]

    Yilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi, Linyong Nan, Lyuhao Chen, Yixin Liu, Xiangru Tang, Rui Zhang, and Arman Cohan. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.852 D oc M ath-eval: Evaluating math reasoning capabilities of LLM s in understanding long and spec...

    2024 doi

  22. [30]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. https://doi.org/10.18653/v1/2021.acl-long.254 TAT - QA : A question answering benchmark on a hybrid of tabular and textual content in finance . In Proceedings of...

    2021 doi

  23. [31]

    Fengbin Zhu, Ziyang Liu, Fuli Feng, Chao Wang, Moxin Li, and Tat Seng Chua. 2024. https://doi.org/10.1145/3677052.3698685 Tat-llm: A specialized language model for discrete reasoning over financial tabular and textual data . In Proceedings of the 5th ACM International Conferen...

    2024

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.