{"id":"380c6b12-587e-4a18-a525-fa6ca13a56e6","arxiv_id":"2412.11757","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SciTaT is a 953-question benchmark requiring joint reasoning over scientific tables and text across four reasoning types, on which the proposed CAR pipeline outperforms standard prompting baselines.","lead":"This paper introduces SciTaT, a new question answering benchmark that asks models to reason over both scientific tables and text, with questions covering look-up, numerical, data analysis, and tabulation types. It also proposes CAR, a baseline that combines a calculator and a reasoner, and reports that CAR improves average scores by about 12.9 percent over simpler prompting methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-label validity in SciTaT rests on an unmeasured post-hoc gpt-4o refinement step; the contradictory generation prompt in Appendix B.1 undermines the claim that every question requires joint table+text reasoning.","rationale":"The reader's weakest assumption (gold-label accuracy after light human refinement) is real and is part of what I flag, but my strongest concern is more targeted: the Appendix B.1 prompt's \"answerable based on the paragraph alone\" instruction contradicts the main text's joint-reasoning requirement (Figure 3), directly threatening the claim that SciTaT covers table-and-text joint reasoning at all. This is an internal inconsistency, not just missing statistics. The concrete test of answering TaT questions with tables removed would settle whether the contradiction produced actual data bias. I also note the reader's other points: no inter-annotator agreement, no error rates, single-run baselines with no error bars, and a 12.9% average improvement over all metrics without CIs. These support CONDITIONAL rather than REJECT because the underlying resource may still be useful and the issues are empirically addressable. I keep the verdict CONDITIONAL and recommend the authors run the proposed checks before final acceptance. The paper does disclose compensation, training, and two-round validation, which is good, but it does not quantify agreement or post-refinement accuracy. The prompt contradiction is the strongest internal evidence that the central claim of joint reasoning is insecure.","tokens_in":19047,"tokens_out":1561,"duration_ms":13819,"concrete_test":"Download the released SciTaT data and prompts. For a random sample of at least 100 questions from the TaT source category, hide the tables and ask an LLM (or human annotator) to answer from the paragraph alone; also hide the paragraph and answer from tables alone. If more than a small fraction (e.g., >10%) are fully answerable without the table, the \"requires joint reasoning\" claim fails and the benchmark's core novelty is weakened. Additionally, re-run Table 5's CAR-vs-CoT comparison with 20 random seeds or temperature-0 samples and report mean and bootstrap 95% CI: if the 12.9% average improvement has overlapping CIs on any metric, the headline effectiveness claim needs qualification.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that SciTaT requires joint reasoning over tables and text is load-bearing for the benchmark's novelty, yet the Appendix B.1 question-generation prompt contains a direct contradiction: it states the question \"must be answerable based on the paragraph alone, and not answerable only based on the table.\" This instruction conflicts with the stated requirement in the main text (Figure 3) that \"the question is best answered by referring to both the tables and the text simultaneously.\" If the generation prompt was used as printed, a large fraction of the 953 questions may be answerable from the paragraph alone, meaning the 331 TaT questions (Table 3) are not verified to require joint reasoning. The paper reports no analysis of how often the annotator-selected answer source was changed during refinement, no inter-annotator agreement, and no post-refinement error rate (Sections 2.3–2.4). Since the benchmark's value and the 12.9% CAR improvement are both measured against these gold labels, systematic label error or a systematic bias toward paragraph-only questions would directly invalidate the headline comparison with HybridQA/TAT-QA/SPIQA. The paper claims answers were corrected by humans, but the extent, error type, and residual disagreement are unreported, so the gold standard is currently unquantified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SCITAT, a question-answering benchmark for scientific tables and text, containing 953 questions derived from 871 arXiv papers in AI/ML. The benchmark covers four reasoning types (Look Up, Numerical Reasoning, Data Analysis, Tabulation) with 13 subtypes, and includes two evaluation settings, long-context and short-context. The authors also propose a baseline method, CAR, which combines a Calculator module (that generates and executes code to extract numerical information) with a Reasoner module (that performs final reasoning), reporting an average improvement of 12.9% over direct QA, CoT, and PoT baselines. Dataset construction relies on gpt-4o-generated questions, rationales, and answers, followed by human refinement and two-round validation. The paper also provides error analysis and discusses challenges such as grounding, complex calculation, and domain knowledge.","tokens_in":19301,"tokens_out":6109,"duration_ms":54588,"significance":"If the validity of the gold labels and the joint table-text requirement are established, SCITAT would fill a real gap in scientific QA by including Data Analysis and Tabulation reasoning types that are rare in existing datasets, and by including a substantial fraction (331/953, about 35%) of questions whose answer sources are both tables and text. The proposed CAR baseline is a reasonable first attempt at combining program-aided computation with free-form reasoning, and the authors publicly release their code and data. The main results, including the 12.9% average improvement and the finding that gpt-4o still scores below 50% EM/F1, would be useful to the community. However, the current manuscript does not provide sufficient evidence for the benchmark's core properties, namely the joint-reasoning requirement and the correctness/unbiasedness of the gold labels, and the experimental claims lack statistical grounding.","major_comments":[{"comment":"The question-generation prompt in Appendix B.1 states that the question 'must be answerable based on the paragraph alone, and not answerable only based on the table,' which directly contradicts the requirement in Figure 3 that 'the question is best answered by referring to both the tables and the text simultaneously.' This contradiction is load-bearing because the 331 TaT questions in Table 3 are the basis for the claim that SCITAT requires joint reasoning over tables and text. If the printed prompt was used as-is, many generated questions may be answerable from the paragraph alone, and the annotator-selected answer source (Table 3) would not verify that both sources are actually needed. The authors must resolve this discrepancy and provide evidence that the 331 TaT questions truly require joint reasoning, for example by reporting how often human refinement changed the answer source, or by conducting a post-hoc verification on a sample that the paragraph alone is insufficient.","section":"Section 2.2 and Appendix B.1 (Table 8)"},{"comment":"The paper reports no inter-annotator agreement, no post-refinement error rate, and no quantitative measure of how much human editing altered the gpt-4o-generated questions, rationales, and answers. Since the gold labels are the yardstick for all model evaluations and for the headline 12.9% improvement, the absence of such quality metrics leaves the validity of the benchmark unquantified. The authors should report IAA (e.g., Cohen's kappa on answer-source selection and on answer correctness), a sampled post-refinement error rate (e.g., on a held-out set independently re-annotated), and statistics on the fraction of instances whose question, rationale, or answer was changed during refinement.","section":"Sections 2.3 and 2.4"},{"comment":"All experimental results are reported as single runs with no variance or significance testing. The claim that CAR improves over baselines by an average of 12.9% could be within run-to-run noise given the inherent stochasticity of LLM prompting. The authors should provide multiple runs (with standard deviations) for at least the main comparisons, and ideally paired significance tests (e.g., bootstrap or approximate randomization) to support the conclusion that CAR's improvements are systematic.","section":"Section 4.2, Table 5"},{"comment":"The evaluation protocol is ambiguous: the text says short-form answers are evaluated with EM and free-form answers with F1/BERTScore, but Table 5 reports a single set of EM, F1, and BERTScore per setting without stating the subset each metric is computed on. Figure 5's statement that F1 and BERTScore are 0 for Look Up (all short-form) and EM is 0 for Data Analysis (all free-form) suggests that metrics are computed over all questions, which would mean free-form answers automatically receive EM 0 and short-form answers automatically receive F1 0. The authors must clarify the exact aggregation rule, the denominators for each metric (number of short-form vs. free-form questions), and how the 12.9% average improvement is computed across these heterogeneous metrics.","section":"Section 4.1.1 and Table 5"}],"minor_comments":[{"comment":"The description of how the 650 SparkRA questions were categorized into reasoning types is brief; please specify whether this categorization was performed by the authors or by annotators, and whether any agreement measure was used for the categorization.","section":"Section 2.2"},{"comment":"The paper does not report the number of papers initially collected, how many were removed by the filtering criteria (papers without tables, irrelevant paragraphs, etc.), and the final yield rate; including these statistics would improve the transparency of dataset construction.","section":"Section 2.1"},{"comment":"The comparison tables use check marks and crosses for reasoning types, but Table 7 does not include the same set of datasets as Table 1, and the meaning of 'R' (Rationale) in Table 7 is not defined in the caption; please make the comparison fully consistent and self-explanatory.","section":"Table 1 and Table 7"},{"comment":"The annotation statistics report 10 annotators for the first round and 2 for validation, but the earlier description in Section 2.4.1 mentions a training requirement with an Exact Match threshold of 95%; please clarify how many annotators passed the training and whether the two-round validation was performed by the same or different annotators across instances.","section":"Appendix C.2"},{"comment":"The limitations section lists language and single-turn scope, but does not mention the potential limitation of reliance on gpt-4o-generated questions or the lack of external validation of the gold labels; given that the benchmark's validity is central, this should be acknowledged.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the contradictory generation prompt is well-founded and should be addressed head-on; it is the kind of issue that a careful internal consistency check should have caught before submission. The manuscript is otherwise a useful contribution to scientific QA, but the missing validation statistics and the ambiguity in the evaluation protocol currently prevent a reliable assessment of the benchmark's value. I would encourage the editor to request a revision that resolves these points rather than rejecting, as the underlying resource appears potentially valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark is worth a look. SciTaT fills a genuine gap: scientific QA that forces joint reasoning over tables and text, with reasoning types beyond lookup and arithmetic. As far as I can tell from the comparisons, it is the first SQA set to include Data Analysis and Tabulation, and 331 of the 953 questions are explicitly labeled as requiring both table and text. The construction process is described in enough detail to reproduce. The authors use gpt-4o for initial generation, then graduate-student annotators refine and validate in two rounds. The CAR baseline is an incremental pipeline—Calculator (program-of-thought) feeding a Reasoner (chain-of-thought)—but the results are consistent across Llama3.1-8B/70B and gpt-4o, and the ablation supports the design. Giving credit where it is due, the paper ships prompts, data, and error analysis.\n\nNow the soft spots. The first is real. The generation prompt in Appendix B.1, Table 8, says the question \"must be answerable based on the paragraph alone, and not answerable only based on the table.\" That directly contradicts the main-text Figure 3 requirement that \"the question is best answered by referring to both the tables and the text simultaneously.\" If the prompt was used as printed, gpt-4o would generate paragraph-only questions, and the 331 TaT instances would not be verified as joint. The authors never mention this discrepancy, and they do not report how often annotators changed the answer source during refinement. This is a load-bearing issue for the novelty claim, not a cosmetic one.\n\nSecond, gold-label validity is unquantified. No inter-annotator agreement, no post-refinement error rate, no measure of edit distance between gpt-4o output and final answer. The headline 12.9% improvement is measured against these labels, so the whole evaluation rests on an unmeasured quality step.\n\nThird, the results are single-run with no variance or significance testing. Minor for a benchmark paper, but treat the improvement as a point estimate.\n\nWho should read this: anyone building scientific QA systems or designing benchmarks. It deserves peer review, not desk rejection. Send it back for major revision: resolve the prompt contradiction, add reliability statistics, and release a sample of post-refinement annotations. If they do that, SciTaT becomes a solid community resource.","headline":"SciTaT targets a real gap in scientific QA, but the contradictory generation prompt and unmeasured gold-label quality mean the joint-reasoning claim is not yet established.","tokens_in":19840,"tokens_out":5455,"would_cite":false,"duration_ms":50374,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"953-question benchmark tests joint table-and-text reasoning","keywords":["scientific question answering","table-and-text reasoning","QA benchmark","reasoning taxonomy","large language models","data analysis","tabulation","chain-of-thought"],"falsifier":"Take a stratified random sample of SciTaT instances and have independent domain-expert annotators answer each question blind to the published gold answer, then measure agreement and the proportion of questions that are actually answerable from their context; if a substantial share (say more than 10%) of gold answers are wrong or unanswerable, the benchmark's central validity claim collapses. A second check is to rerun CaR against chain-of-thought over multiple random seeds and check whether the 12.9% average improvement persists.","tokens_in":18887,"feed_emoji":"📊","tokens_out":6596,"duration_ms":55417,"temperature":0.7,"pith_summary":"Scientific question answering stops at reading either text or tables; this paper argues that the real questions researchers ask cut across both and demand a wider spread of reasoning moves. It introduces SciTaT, 953 questions drawn from 871 arXiv papers, organised into 4 reasoning types and 13 subtypes that include look-up, numerical reasoning, data analysis, and tabulation. The paper's central claim is that this benchmark is harder and more realistic than existing scientific QA datasets because over a third of its questions require combining information from tables and text in the same answer. To show the benchmark is usable, the authors build CaR, a baseline that first computes numerical information with code and then reasons over it, reporting an average 12.9% improvement over direct, chain-of-thought, and program-of-thought prompting. If the benchmark is sound, the field gains a stress test for the kinds of analytical questions working scientists actually ask.","feed_headline":"953-question benchmark tests joint table-and-text reasoning","feed_subtitle":"Best tested AI systems still score below 50 percent on questions that combine tables and prose.","key_machinery":"The load-bearing object is the SciTaT benchmark itself, with its four-by-thirteen reasoning taxonomy and its enforced table-and-text contexts. The taxonomy is the mechanism that differentiates the benchmark: it was induced from 650 questions sampled from a researcher-facing QA platform plus reasoning types from earlier datasets, and each subtype has a prescribed question format and evaluation behavior. The other mechanism is the CaR baseline, which splits the task into a Calculator and a Reasoner: the Calculator is prompted to write and run code that returns a complete sentence of numerical information instead of a bare answer, and the Reasoner performs chain-of-thought reasoning over that information while explicitly reflecting on whether the information is correct. The combination is what produces the paper's claimed 12.9% average improvement, and the ablation shows removing either module or reversing their order degrades performance.","core_discovery":"The central claim is that realistic scientific question answering must force models to reason jointly over tables and text, and that this requirement can be measured with a benchmark whose reasoning types come from real researcher queries rather than dataset convenience. The paper develops SciTaT, a 953-question benchmark from 871 papers, with four reasoning types and 13 subtypes, including Data Analysis and Tabulation that earlier datasets mostly omit. Each instance pairs a question with a rationale and an answer source labelled Text, Table, or Table-and-Text, and more than one-third of the questions require both modalities. On this resource, gpt-4o and Llama3.1 models all fall short of 50% exact match and F1 under the best baseline, and the paper takes this as evidence that SciTaT measures something current systems cannot yet do reliably.","pith_inferences":["A natural extension the paper does not run is a human re-annotation study: because the gold labels originate from gpt-4o, an independent blind re-answer of a random sample would quantify label noise and show how much of the 12.9% gap is real model behaviour versus annotation bias.","The reasoning taxonomy could be transplanted to other scientific domains, such as biomedicine or chemistry, where the same four reasoning families should recur; a testable prediction is that data-analysis and tabulation questions remain the hardest there as well.","Since the questions were generated to require both tables and text, SciTaT likely rewards models that explicitly fuse the two modalities; comparing CaR against a retrieval-augmented model in the long-context setting would isolate whether the gain comes from locating evidence or from computing numbers."],"forward_implications":["Even the strongest tested pipeline, CaR with gpt-4o, stays below 50% exact string match on short answers and below 50% F1 on free-form answers, so SciTaT marks current LLM performance as insufficient for scientific analysis questions.","Data Analysis is the hardest reasoning type, and Tabulation produces high lexical overlap with the gold answer but near-zero exact match, suggesting tabulation needs a different evaluation metric than string equality.","CaR's margin over chain-of-thought is larger in the long-context whole-paper setting, which points to numerical pre-extraction as a way to reduce the burden of grounding in long scientific documents.","Free-form answers and short-form answers fail in different ways, including omissions, grounding errors, and redundancy, so progress on SciTaT likely requires separate treatments for the two answer regimes."],"supporting_citations":[{"why":"Supplies the researcher-facing QA platform whose 650 questions are sampled to induce the paper's reasoning-type taxonomy.","marker":"Wu et al., 2024a"},{"why":"Contributes the SciTab benchmark whose reasoning types are combined with real-query types to form the 13 subtypes.","marker":"Lu et al., 2023"},{"why":"Provides TableBench, an additional source of reasoning types and a reference for the taxonomy and baselines.","marker":"Wu et al., 2024b"},{"why":"SPIQA is the prior multimodal scientific QA dataset SciTaT compares against, and its prompting style is used as the Direct QA baseline.","marker":"Pramanick et al., 2024"},{"why":"Chain-of-thought prompting is the strongest single baseline and the reasoning mechanism inside the Reasoner.","marker":"Wei et al., 2022"},{"why":"Program-of-thought methods are the baseline family that the Calculator adapts, with the twist that code returns a sentence of numerical information rather than a direct answer.","marker":"Gao et al., 2023; Chen et al., 2023"},{"why":"gpt-4o generates the initial questions, rationales, and answers and is the strongest evaluated model.","marker":"OpenAI et al., 2024"},{"why":"Llama3.1-Instruct is the open-source model used for main and ablation experiments.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["AI fails half of joint table-text science questions in new benchmark","SciTaT benchmark stumps top models on table-text reasoning","953-question benchmark demands integrated table-text reasoning","New benchmark shows AI under 50% on mixed table-text QA","Table-text reasoning gap exposed by 953-question SciTaT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that gpt-4o-drafted questions and answers, after light human refinement, are accurate and unbiased enough to serve as gold labels, yet the paper reports no inter-annotator agreement, no post-refinement error counts, and no record of how much editing changed the drafts.","fun_headline_variants_meta":{"raw":{"variants":["AI fails half of joint table-text science questions in new benchmark","SciTaT benchmark stumps top models on table-text reasoning","953-question benchmark demands integrated table-text reasoning","New benchmark shows AI under 50% on mixed table-text QA","Table-text reasoning gap exposed by 953-question SciTaT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1399,"prompt_tokens":866,"completion_tokens":533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":482,"tokens_out":533,"duration_ms":4890,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:36:26.366585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a stratified random sample of SciTaT instances and have independent domain-expert annotators answer each question blind to the published gold answer, then measure agreement and the proportion of questions that are actually answerable from their context; if a substantial share (say more than 10%) of gold answers are wrong or unanswerable, the benchmark's central validity claim collapses. A second check is to rerun CaR against chain-of-thought over multiple random seeds and check whether the 12.9% average improvement persists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SPIQA is the prior multimodal scientific QA dataset SciTaT compares against, and its prompting style is used as the Direct QA baseline."}],"review_version":1}