Pith. sign in

REVIEW 4 major objections 6 minor 20 references

Exploring Performance Contrasts in TableQA: Step-by-Step Reasoning Boosts Bigger Language Models, Limits Smaller Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Step-by-step TableQA prompting lifts models above 70B and sinks models around 7B.

desk verdict The headline contrast is real but overstated; the diagnostic sub-task and replacement experiments are the real contribution. read the letter →

arxiv 2411.16002 v1 pith:2WBLMH26 submitted 2024-11-24 cs.CL

Haoyan Yang , Yixuan Wang , Keyue Tong , Hongjin Zhu , Yuanxin Zhang This is my paper · ORCID
classification cs.CL
keywords TableQAstep-by-stepreasoningpromptingmethodsmodelscalelanguagemodelsstructureunderstandingChain-of-Thoughterrorpropagation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a step-by-step prompting flow for table question answering, called Table-Logic, helps language models with 70B or more parameters but hurts models around 7B parameters. On HybridQA, the flow lifts Llama-3-70B by 7.8 percentage points while dropping Llama-2-7B by 11 points, and a similar sign flip appears across most datasets and models. The authors trace the failure to early errors: small models misidentify critical columns and rows, and every later step inherits those errors. Their sub-task experiments show the largest capability gap between bigger and smaller models is in column-related table-structure skills. If the pattern holds, it means step-by-step reasoning methods should not be applied to small models without fixing their table-structure understanding first.

What carries the argument

The load-bearing machinery is the Table-Logic sequential prompt chain, a five-step decomposition in which each prompt receives the previous step's output: parse the table dictionary, name critical columns, name critical rows, name any aggregation/calculation/comparison, then produce the final answer with all intermediate choices pasted back in. Around this flow, the paper builds seven diagnostic sub-tasks (count rows, count columns, row finding, column finding, value lookup, replacement of critical rows/columns, replacement of aggregation) to measure capabilities separately. The sub-tasks do the explanatory work: they show that the biggest measured shortfall of smaller models is in table-structure operations—especially column identification—which are exactly the early steps whose errors propagate through the rest of the chain.

What would settle it

Run Llama-3-70B and Llama-2-7B on WikiTableQuestions under vanilla and Table-Logic prompts, scoring with a deterministic exact-match metric across multiple independent runs; if the sign flip between the two models disappears or reverses, the claimed contrast depends on the GPT-4 judge rather than on the reasoning method.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the same structured reasoning flow reverses its effect depending on model scale. The Table-Logic flow—identify critical columns, then critical rows, then required aggregations, then answer—improves accuracy on bigger LMs in nearly every setting (best: +7.8% on HybridQA for Llama-3-70B) but degrades smaller LMs in most settings (worst: -11% on HybridQA for Llama-2-7B). Through seven sub-tasks the paper localizes the cause: bigger and smaller models differ most on counting columns and finding a column from a value (up to 78% gap), while question-side skills such as aggregation are closer. Smaller models therefore begin the reasoning chain with wrong structural anchors, and the cascade of wrong intermediate outputs explains why the very same prompts that amplify big-model performance suppress small-model performance.

Load-bearing premise

The entire contrast rests on accuracy numbers produced by a single GPT-4 yes/no judge, and some of the headline differences are as small as 0.1 percentage points, so if the judge is noisy or biased the reported sign flip could be overstated.

Editorial extensions

If this is right

  • For ~7B models, plain answering beats step-by-step prompting on most TableQA settings, so deployment choices should treat multi-step reasoning as a risk until table-structure skills improve.
  • Bigger models gain more on Wikipedia-style hybrid tables (HybridQA, WikiTQ) than on financial tables (TAT-QA), suggesting gains concentrate in tasks where column disambiguation matters most.
  • Because early steps gate later ones, improving column identification should produce larger downstream accuracy gains than improving row finding or aggregation.
  • Table-Logic outperforms Self-Augmentation on most bigger-model settings, indicating that a more decomposed prompt with explicit intermediates is the better chain for large models.
  • Accuracy gains from step-by-step TableQA methods should be reported with the model scale stated, since the same method can show a double-digit gain and a double-digit loss across scales.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the early column/row steps of a small model were replaced by outputs from a bigger model or a cheap rule-based selector, the remaining steps might close most of the gap; this is a direct test of the error-propagation story and is not run in the paper.
  • The same scale-dependent sign flip may appear in other structured tasks such as SQL generation or form-filling, where the first parsing step is structurally hard; applying Table-Logic-style decomposition to those tasks would test the generality of the pattern.
  • The paper's table-structure gap results (e.g., column finding 0.207 vs 0.717) suggest a targeted fine-tuning intervention on column identification for ~7B models could be a cheap, high-leverage fix; the authors do not propose such an intervention.
  • The GPT-4 judge's 93.6% agreement was reported on 800 examples with no per-dataset breakdown; if judge errors correlate with longer intermediate outputs, the magnitude—and possibly the direction—of the small-model contrast could change under a stricter exact-match metric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Table-Logic, a sequential prompting method for Table Question Answering that decomposes the task into identifying critical columns and rows, then determining required aggregations/calculations/comparisons before producing the final answer. It evaluates this method against vanilla prompting and Self-Augmentation on TAT-QA, HybridQA, and WikiTQ, using three larger LMs (Llama-3-70B, GPT-3.5-Turbo, Qwen-1.5-72B) and three smaller LMs (Llama-2-7B, Vicuna-7B, Qwen-1.5-7B). The reported results show consistent accuracy gains for larger models in all nine settings and declines in eight of nine settings for smaller models. The authors then design seven sub-tasks to compare capabilities between model sizes, concluding that smaller LMs lack table-structure understanding. The central claim is that step-by-step reasoning methods are unsuitable for smaller LMs.

Significance. The paper makes a useful, falsifiable observation about model-size-dependent effects of decomposed prompting in TableQA. The breadth of models and datasets is a strength, and the code is public. If the central contrast is statistically robust, the finding would be practically relevant: practitioners should not expect step-by-step pipelines to transfer from large to small models. The sub-task analysis is a first step toward localizing the deficit, but its self-referential design and missing details limit its value. The main weakness is that all accuracy numbers depend on a single GPT-4 judge with no error bars, so the strength of the central claim currently exceeds the evidence.

major comments (4)
  1. [Section 4.4 and Table 1] The accuracy deltas that support the 'limits smaller LMs' claim are measured with a single GPT-4 judge, and the paper reports no confidence intervals, significance tests, or repeated decoding runs. Several decisive cells are within plausible evaluation noise, notably Qwen-1.5-7B on WikiTQ (0.434 vs 0.433, a -0.1 pp difference) and Llama-2-7B on WikiTQ (0.412 vs 0.403, -0.9 pp). The one-sentence report of 93.6% human agreement on 800 examples lacks a per-dataset and per-answer-format breakdown, so systematic judge bias against small-model outputs cannot be ruled out. As a result, the universal conclusion that step-by-step reasoning is unsuitable for smaller LMs is not statistically supported.
  2. [Sections 5.1 and Abstract] The paper generalizes from two specific protocols (Table-Logic and Self-Augmentation) to 'step-by-step reasoning methods' as a class. This is an overgeneralization because the tested methods are both multi-call prompting pipelines; single-pass chain-of-thought prompting, the most common step-by-step variant, is not evaluated. The claim should be restricted to 'the tested decomposed prompting protocols' unless further variants are included.
  3. [Section 5.2.1 and Table 3] The statement that 'almost all smaller LMs improve' after receiving the three information types is not supported by the table, which shows several declines: Vicuna-7B on HybridQA with table-structure information drops from 0.300 to 0.294, and Qwen-1.5-7B on HybridQA drops from 0.603 to 0.586 with table-structure information and from 0.603 to 0.592 with aggregation information. The text does not specify the exact contents of the 'With Table Structure', 'With Column and Row', and 'With Aggregation' conditions, so the reasons for these negative cases are unclear and the claim of a widespread capability deficit needs a more nuanced analysis.
  4. [Section 3.2 and Table 2] The sub-task analysis is not reproducible because the paper omits the number of examples per sub-task, the evaluation prompt, and the operational definitions of the 'replacement' tasks. In addition, Section 5.2.2 states the largest gap is in column finding 'reaching 78%', but Table 2 reports an absolute gap of 0.510, and the relative gap is approximately 71%; the paper should state which normalization is used. These omissions undermine the explanatory conclusions that are presented as the third contribution.
minor comments (6)
  1. [Section 1] The definition of bigger LMs as having more than 70B parameters is contradicted by the inclusion of GPT-3.5-Turbo, whose parameter count is undisclosed and likely below that threshold; please clarify the grouping criterion.
  2. [Section 4.4] The differences in Table 1 are reported in percentage points but labeled as percentages; add a footnote to avoid confusion, and specify whether GPT-4 evaluation used greedy decoding or a fixed temperature.
  3. [Table 3] The table is formatted incorrectly in the manuscript, with missing or misaligned column headers; please fix the layout.
  4. [Section 5.2.2] The '78%' gap for column finding should be derived explicitly from the numbers in Table 2 or from the underlying per-dataset results.
  5. [References] Several citations appear as 'et al., 2023a' and 'et al., 2023b' without the first author name; use standard author-year citations.
  6. [Appendix A] In step 4 of the prompt, there is a typo: 'Critical columns: column' should read 'Critical columns: {column}'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline result is an external empirical comparison, not a fitted or definitionally forced prediction.

full rationale

The paper's central claim is an empirical observation from Table 1: Table-Logic improves accuracy for three larger LMs in most dataset/model cells and degrades it for three smaller LMs in most cells. No parameter is fitted to the outcome, no quantity is defined in terms of the quantity it is said to predict, and no prior result by these authors is cited as the load-bearing justification for the main contrast. The accuracy measurements use an external GPT-4 judge with a reported 93.6% agreement against human evaluation on 800 sampled examples, so the evaluation is not definitionally identical to the paper's own prompting method. The seven sub-tasks are derived from the method's own decomposition, and the replacement experiments apply smaller-LM intermediate outputs to bigger-LM reasoning; this is a self-referential diagnostic design, but it is used to characterize capability gaps rather than to construct the headline accuracy numbers. The stated limitations (cost, ignored interaction effects) are acknowledged limitations, not circularity. Concerns about judge noise, small differences such as -0.1 or +0.4 percentage points, and lack of confidence intervals are evaluation-validity concerns, not circularity: they do not show that any reported result reduces by construction to its inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters and no invented entities. The empirical claims rest on the three domain assumptions above, chiefly the reliability of GPT-4 as judge and the interpretability of the big-vs-small comparison as a scale effect.

assumptions (3)
  • domain assumption GPT-4-based yes/no evaluation yields accuracy measurements that are unbiased and precise enough to compare models, validated on 800 examples with 93.6% agreement with human judgments.
    All accuracy numbers in Table 1 and the sub-task results depend on this judge; a bias toward or against long step-by-step outputs could create the reported contrast. Section 4.4.
  • domain assumption The grouping of models into >70B and ~7B isolates model scale as the relevant variable, setting aside model family, training recency, and instruction tuning.
    The bigger set includes Llama-3-70B and GPT-3.5-Turbo while the smaller set includes Llama-2-7B and Vicuna-7B, so the observed contrast could partly reflect version or fine-tuning differences rather than size alone. Section 4.2.
  • ad hoc to paper The seven sub-tasks derived from Table-Logic's own steps decompose the TableQA capability into the dimensions that explain the performance gap.
    The sub-task taxonomy is constructed from the authors' method (Section 3.2), so it may omit other relevant abilities such as numerical reasoning or text-table alignment, and the replacement tests assume the bigger model's intermediate outputs are a valid oracle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Performance Contrasts in TableQA: Step-by-Step Reasoning Boosts Bigger Language Models, Limits Smaller Language Models." pith.science (2026). https://pith.science/paper/2WBLMH26

@misc{pith2026241116002,
  author       = {Pith},
  title        = {Pith review of: Exploring Performance Contrasts in TableQA: Step-by-Step Reasoning Boosts Bigger Language Models, Limits Smaller Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WBLMH26}},
  note         = {Machine review of arXiv:2411.16002}
}
read the original abstract

This paper proposes a detailed prompting flow, termed Table-Logic, to investigate the performance contrasts between bigger and smaller language models (LMs) utilizing step-by-step reasoning methods in the TableQA task. The method processes tasks by sequentially identifying critical columns and rows given question and table with its structure, determining necessary aggregations, calculations, or comparisons, and finally inferring the results to generate a precise prediction. By deploying this method, we observe a 7.8% accuracy improvement in bigger LMs like Llama-3-70B compared to the vanilla on HybridQA, while smaller LMs like Llama-2-7B shows an 11% performance decline. We empirically investigate the potential causes of performance contrasts by exploring the capabilities of bigger and smaller LMs from various dimensions in TableQA task. Our findings highlight the limitations of the step-by-step reasoning method in small models and provide potential insights for making improvements.

Figures

Figures reproduced from arXiv: 2411.16002 by the authors.

Figure 1
Figure 1. Comparative illustration of step-by-step rea [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Table-Logic Sequential Prompting methodology and a breakdown of capabilities between [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Radar chart displaying the table-related capabilities of big￾ger and smaller LMs across three datasets averagely [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 linked inside Pith

  1. [1]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  2. [2]

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. 2021. https://arxiv.org/abs/2004.07347 Hybridqa: A dataset of multi-hop question answering over tabular and textual data . Preprint, arXiv:2004.07347

    arXiv 2021

  3. [3]

    Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. 2024. https://arxiv.org/abs/2402.18312 How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning . Preprint, arXiv:2402.18312

    arXiv 2024

  4. [4]

    Hugo Touvron et al. 2023 a . https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . Preprint, arXiv:2307.09288

    arXiv 2023

  5. [5]

    Rohan Anil et al. 2023 b . https://arxiv.org/abs/2305.10403 Palm 2 technical report . Preprint, arXiv:2305.10403

    arXiv 2023

  6. [6]

    Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. https://arxiv.org/abs/2210.11610 Large language models can self-improve . Preprint, arXiv:2210.11610

    arXiv 2022

  7. [7]

    Nengzheng Jin, Joanna Siebert, Dongfang Li, and Qingcai Chen. 2022. A survey on table question answering: recent advances. In China Conference on Knowledge Graph and Semantic Computing, pages 174--186. Springer

    2022

  8. [8]

    Fangyu Lei, Tongxu Luo, Pengqi Yang, Weihao Liu, Hanwen Liu, Jiahe Lei, Yiming Huang, Yifan Wei, Shizhu He, Jun Zhao, and Kang Liu. 2023. https://arxiv.org/abs/2310.15075 Tableqakit: A comprehensive and practical toolkit for table-based question answering . Preprint, arXiv:2310.15075

    arXiv 2023

Show all 20 references
  1. [9]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://arxiv.org/abs/2303.16634 G-eval: Nlg evaluation using gpt-4 with better human alignment . Preprint, arXiv:2303.16634

    2023 arXiv

  2. [10]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  3. [11]

    Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305

    2015 arXiv

  4. [12]

    Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 645--654

    2024

  5. [13]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models . Preprint, arXiv:2203.11171

    2023 arXiv

  6. [14]

    Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024. https://arxiv.org/abs/2401.04398 Chain-of-table: Evolving tables in the reasoning chain for table...

    2024 arXiv

  7. [15]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    2023 arXiv

  8. [16]

    Yilun Zhao, Haowei Zhang, Shengyun Si, Linyong Nan, Xiangru Tang, and Arman Cohan. 2023. https://arxiv.org/abs/2305.14987 Investigating table-to-text generation capabilities of llms in real-world information seeking scenarios . Preprint, arXiv:2305.14987

    2023 arXiv

  9. [17]

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. https://arxiv.org/abs/2205.10625 Least-to-most prompting enables complex reasoning in large language models . Preprint, arXiv...

    2023 arXiv

  10. [18]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624

    2021 arXiv

  11. [19]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  12. [20]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.