{"id":"2f7e13ec-ceeb-4a0e-945d-1d863e2508fe","arxiv_id":"2411.16002","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A sequential prompting flow for TableQA improves accuracy in 70B-class LLMs but lowers it in most 7B-class LLMs, with the largest diagnosed gap in table-structure understanding.","lead":"This paper introduces Table-Logic, a step-by-step prompting method for answering questions over tables, and finds it improves 70B-parameter language models but degrades most 7B models. It then decomposes TableQA into seven sub-tasks to show that small models mainly lack table-structure understanding, like counting columns and finding headers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central contrast rests on accuracy differences measured by a single GPT-4 judge with no confidence intervals or repeated runs; several supporting differences (e.g., -0.1%, -0.9%, +0.4%) are within plausible evaluation noise, so the universal \"limits smaller LMs\" claim is not yet established.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the accuracy measurements rely on a single GPT-4 judge with no confidence intervals or repeated runs, and differences as small as 0.1 percentage points are used to support the central claim. I agree with this assessment. The broad direction of the result—bigger models tend to benefit, smaller models tend to be hurt—is plausible and is corroborated by several large differences, so an outright rejection is not warranted. However, the paper's abstract and Section 5.1 state that step-by-step reasoning methods are \"unsuitable for smaller LMs\" without qualification, yet Table 1 contains small-model cells where T-Logic slightly improves accuracy (Llama-2-7B on TAT-QA, +0.4 pp) or is essentially neutral (Qwen-1.5-7B on WikiTQ, -0.1 pp). Without statistical reliability, these cells could flip sign, meaning the universal negative claim is overreaching. The most direct fix is to report confidence intervals or repeated runs and to soften the conclusion to a tendency rather than a universal limit. This does not change the reader's conditional verdict; it reinforces that the paper should be accepted only with these additions or revisions. Other potential concerns, such as the confounding of model size with model family and generation, are partially mitigated by the same-family Qwen-1.5-72B vs Qwen-1.5-7B comparison, though that comparison also shows a near-zero effect on WikiTQ. The sub-task analysis is secondary and does not bear directly on the headline claim. Thus, the central load-bearing issue remains the absence of uncertainty quantification around the accuracy measurements, which the reader correctly identified.","tokens_in":7201,"tokens_out":6561,"duration_ms":60588,"concrete_test":"Re-run the three small-model experiments on WikiTQ and TAT-QA with at least 5 independent decoding seeds (or verify that temperature was 0), evaluate each run with the same GPT-4 judge, and compute bootstrap confidence intervals or paired significance tests for the T-Logic-minus-vanilla differences per model and dataset. Separately, compute the judge-versus-human agreement per dataset on a stratified sample of 200 examples per dataset, and test whether agreement is lower for small-model outputs than for big-model outputs. If the -0.1% and -0.9% cells flip sign or their 95% CIs include zero, the \"limits smaller LMs\" claim should be qualified to \"tends to reduce accuracy in most, but not all, evaluated settings.\" If the large negative cells remain significant, the core contrast stands but the universal phrasing must be softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claim that step-by-step reasoning methods are unsuitable for smaller LMs depends on Table 1 showing T-Logic and Self-Augmentation accuracy relative to vanilla. The evaluation uses one GPT-4 judge, and the only reliability evidence is a one-sentence report of 93.6% agreement on 800 examples, with no per-dataset breakdown. For smaller LMs, the reported T-Logic differences include -0.1 percentage points (Qwen-1.5-7B, WikiTQ), -0.9 pp (Llama-2-7B, WikiTQ), and +0.4 pp (Llama-2-7B, TAT-QA). With typical test-set sizes in these benchmarks, these differences correspond to only a handful of examples. A single judge with a ~6% disagreement rate could easily flip such cells. No confidence intervals, significance tests, or multiple decoding runs are reported, so small declines are statistically indistinguishable from zero. Even if the large headline examples (+7.8%, -11%) are robust, the universal wording \"limits smaller LMs\" is not supported by cells that may be noise. Additionally, the 93.6% human agreement is not broken down by dataset or by predicted-answer format, so a systematic judge bias against small-model outputs (e.g., for terse or partially formatted answers) cannot be ruled out; such a bias would inflate the apparent decline. The central claim would be secure if small-model differences were consistently negative with tight intervals, but the present table does not show that.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Table-Logic, a sequential prompting method for Table Question Answering that decomposes the task into identifying critical columns and rows, then determining required aggregations/calculations/comparisons before producing the final answer. It evaluates this method against vanilla prompting and Self-Augmentation on TAT-QA, HybridQA, and WikiTQ, using three larger LMs (Llama-3-70B, GPT-3.5-Turbo, Qwen-1.5-72B) and three smaller LMs (Llama-2-7B, Vicuna-7B, Qwen-1.5-7B). The reported results show consistent accuracy gains for larger models in all nine settings and declines in eight of nine settings for smaller models. The authors then design seven sub-tasks to compare capabilities between model sizes, concluding that smaller LMs lack table-structure understanding. The central claim is that step-by-step reasoning methods are unsuitable for smaller LMs.","tokens_in":7563,"tokens_out":9236,"duration_ms":70628,"significance":"The paper makes a useful, falsifiable observation about model-size-dependent effects of decomposed prompting in TableQA. The breadth of models and datasets is a strength, and the code is public. If the central contrast is statistically robust, the finding would be practically relevant: practitioners should not expect step-by-step pipelines to transfer from large to small models. The sub-task analysis is a first step toward localizing the deficit, but its self-referential design and missing details limit its value. The main weakness is that all accuracy numbers depend on a single GPT-4 judge with no error bars, so the strength of the central claim currently exceeds the evidence.","major_comments":[{"comment":"The accuracy deltas that support the 'limits smaller LMs' claim are measured with a single GPT-4 judge, and the paper reports no confidence intervals, significance tests, or repeated decoding runs. Several decisive cells are within plausible evaluation noise, notably Qwen-1.5-7B on WikiTQ (0.434 vs 0.433, a -0.1 pp difference) and Llama-2-7B on WikiTQ (0.412 vs 0.403, -0.9 pp). The one-sentence report of 93.6% human agreement on 800 examples lacks a per-dataset and per-answer-format breakdown, so systematic judge bias against small-model outputs cannot be ruled out. As a result, the universal conclusion that step-by-step reasoning is unsuitable for smaller LMs is not statistically supported.","section":"Section 4.4 and Table 1"},{"comment":"The paper generalizes from two specific protocols (Table-Logic and Self-Augmentation) to 'step-by-step reasoning methods' as a class. This is an overgeneralization because the tested methods are both multi-call prompting pipelines; single-pass chain-of-thought prompting, the most common step-by-step variant, is not evaluated. The claim should be restricted to 'the tested decomposed prompting protocols' unless further variants are included.","section":"Sections 5.1 and Abstract"},{"comment":"The statement that 'almost all smaller LMs improve' after receiving the three information types is not supported by the table, which shows several declines: Vicuna-7B on HybridQA with table-structure information drops from 0.300 to 0.294, and Qwen-1.5-7B on HybridQA drops from 0.603 to 0.586 with table-structure information and from 0.603 to 0.592 with aggregation information. The text does not specify the exact contents of the 'With Table Structure', 'With Column and Row', and 'With Aggregation' conditions, so the reasons for these negative cases are unclear and the claim of a widespread capability deficit needs a more nuanced analysis.","section":"Section 5.2.1 and Table 3"},{"comment":"The sub-task analysis is not reproducible because the paper omits the number of examples per sub-task, the evaluation prompt, and the operational definitions of the 'replacement' tasks. In addition, Section 5.2.2 states the largest gap is in column finding 'reaching 78%', but Table 2 reports an absolute gap of 0.510, and the relative gap is approximately 71%; the paper should state which normalization is used. These omissions undermine the explanatory conclusions that are presented as the third contribution.","section":"Section 3.2 and Table 2"}],"minor_comments":[{"comment":"The definition of bigger LMs as having more than 70B parameters is contradicted by the inclusion of GPT-3.5-Turbo, whose parameter count is undisclosed and likely below that threshold; please clarify the grouping criterion.","section":"Section 1"},{"comment":"The differences in Table 1 are reported in percentage points but labeled as percentages; add a footnote to avoid confusion, and specify whether GPT-4 evaluation used greedy decoding or a fixed temperature.","section":"Section 4.4"},{"comment":"The table is formatted incorrectly in the manuscript, with missing or misaligned column headers; please fix the layout.","section":"Table 3"},{"comment":"The '78%' gap for column finding should be derived explicitly from the numbers in Table 2 or from the underlying per-dataset results.","section":"Section 5.2.2"},{"comment":"Several citations appear as 'et al., 2023a' and 'et al., 2023b' without the first author name; use standard author-year citations.","section":"References"},{"comment":"In step 4 of the prompt, there is a typo: 'Critical columns: column' should read 'Critical columns: {column}'.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The GitHub repository name indicates a course project, and the evaluation would need to be substantially strengthened (error bars, per-dataset judge agreement, more rigorous overclaim control) before the paper meets the bar for this journal. The qualitative direction of the main effect is interesting, but the current evidential basis for the universal claim is thin."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's central claim, that step-by-step reasoning helps 70B+ models and hurts 7B models on TableQA, is directionally right and consistent across most model–dataset cells, but the paper oversells it as a universal law. The genuinely useful part is the breakdown: the seven sub-tasks and especially the replacement experiments, where you take the columns/rows/aggregation generated by a small model and feed them into a big model. That's a clean way to localize where the small model's intermediate steps go wrong.\n\nWhat the paper does well: the Table-Logic flow is a reasonable decomposition of TableQA into column selection, row selection, aggregation, and final answer. Showing that this flow improves big models and impairs small ones across three datasets and six models is a solid empirical pattern. The replacement experiments are the strongest evidence in the paper because they directly link the quality of intermediate outputs to final accuracy.\n\nWhere it's soft: the statistics are thin. No confidence intervals, no repeated runs, and the only judge is GPT-4 with a one-sentence human agreement check on 800 examples. Some of the supporting cells are inside noise: Qwen-1.5-7B on WikiTQ goes 0.434 to 0.433 (–0.1 pp), Llama-2-7B on TAT-QA goes 0.522 to 0.526 (+0.4 pp). That is a handful of examples. The paper's own wording, \"step-by-step reasoning methods are unsuitable for smaller LMs,\" is too strong for a pattern where one cell is positive and several are flat. The conclusion should say \"tends to hurt, and sometimes substantially\" rather than \"limits.\"\n\nThe sub-task Table 2 also lacks context: no sample sizes, no per-dataset numbers, so I can't tell whether the big/small gaps are meaningful or just small-sample artifacts. And calling GPT-3.5-Turbo a 'bigger' LM is fine, but for a paper about parameter counts you should hedge since the size is not confirmed.\n\nOverall: the empirical contrast holds up qualitatively, the diagnostics are a worthwhile contribution, and the citation of Wei et al. shows they know the scale-dependence is not entirely new. The paper deserves a serious referee, but it needs error bars, significance tests, and a more careful statement of the conclusion.","headline":"The headline contrast is real but overstated; the diagnostic sub-task and replacement experiments are the real contribution.","tokens_in":8079,"tokens_out":2699,"would_cite":true,"duration_ms":24506,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Step-by-step TableQA prompting lifts models above 70B and sinks models around 7B.","keywords":["TableQA","step-by-step reasoning","prompting methods","model scale","language models","table structure understanding","Chain-of-Thought","error propagation"],"falsifier":"Run Llama-3-70B and Llama-2-7B on WikiTableQuestions under vanilla and Table-Logic prompts, scoring with a deterministic exact-match metric across multiple independent runs; if the sign flip between the two models disappears or reverses, the claimed contrast depends on the GPT-4 judge rather than on the reasoning method.","tokens_in":7026,"feed_emoji":"📊","tokens_out":7739,"duration_ms":63338,"temperature":0.7,"pith_summary":"This paper claims that a step-by-step prompting flow for table question answering, called Table-Logic, helps language models with 70B or more parameters but hurts models around 7B parameters. On HybridQA, the flow lifts Llama-3-70B by 7.8 percentage points while dropping Llama-2-7B by 11 points, and a similar sign flip appears across most datasets and models. The authors trace the failure to early errors: small models misidentify critical columns and rows, and every later step inherits those errors. Their sub-task experiments show the largest capability gap between bigger and smaller models is in column-related table-structure skills. If the pattern holds, it means step-by-step reasoning methods should not be applied to small models without fixing their table-structure understanding first.","feed_headline":"Step-by-step prompts lift big models, sink small ones","feed_subtitle":"Table-Logic flow gains up to 7.8% on 70B+ models while costing smaller models up to 11% accuracy.","key_machinery":"The load-bearing machinery is the Table-Logic sequential prompt chain, a five-step decomposition in which each prompt receives the previous step's output: parse the table dictionary, name critical columns, name critical rows, name any aggregation/calculation/comparison, then produce the final answer with all intermediate choices pasted back in. Around this flow, the paper builds seven diagnostic sub-tasks (count rows, count columns, row finding, column finding, value lookup, replacement of critical rows/columns, replacement of aggregation) to measure capabilities separately. The sub-tasks do the explanatory work: they show that the biggest measured shortfall of smaller models is in table-structure operations—especially column identification—which are exactly the early steps whose errors propagate through the rest of the chain.","core_discovery":"The paper's central discovery is that the same structured reasoning flow reverses its effect depending on model scale. The Table-Logic flow—identify critical columns, then critical rows, then required aggregations, then answer—improves accuracy on bigger LMs in nearly every setting (best: +7.8% on HybridQA for Llama-3-70B) but degrades smaller LMs in most settings (worst: -11% on HybridQA for Llama-2-7B). Through seven sub-tasks the paper localizes the cause: bigger and smaller models differ most on counting columns and finding a column from a value (up to 78% gap), while question-side skills such as aggregation are closer. Smaller models therefore begin the reasoning chain with wrong structural anchors, and the cascade of wrong intermediate outputs explains why the very same prompts that amplify big-model performance suppress small-model performance.","pith_inferences":["If the early column/row steps of a small model were replaced by outputs from a bigger model or a cheap rule-based selector, the remaining steps might close most of the gap; this is a direct test of the error-propagation story and is not run in the paper.","The same scale-dependent sign flip may appear in other structured tasks such as SQL generation or form-filling, where the first parsing step is structurally hard; applying Table-Logic-style decomposition to those tasks would test the generality of the pattern.","The paper's table-structure gap results (e.g., column finding 0.207 vs 0.717) suggest a targeted fine-tuning intervention on column identification for ~7B models could be a cheap, high-leverage fix; the authors do not propose such an intervention.","The GPT-4 judge's 93.6% agreement was reported on 800 examples with no per-dataset breakdown; if judge errors correlate with longer intermediate outputs, the magnitude—and possibly the direction—of the small-model contrast could change under a stricter exact-match metric."],"forward_implications":["For ~7B models, plain answering beats step-by-step prompting on most TableQA settings, so deployment choices should treat multi-step reasoning as a risk until table-structure skills improve.","Bigger models gain more on Wikipedia-style hybrid tables (HybridQA, WikiTQ) than on financial tables (TAT-QA), suggesting gains concentrate in tasks where column disambiguation matters most.","Because early steps gate later ones, improving column identification should produce larger downstream accuracy gains than improving row finding or aggregation.","Table-Logic outperforms Self-Augmentation on most bigger-model settings, indicating that a more decomposed prompt with explicit intermediates is the better chain for large models.","Accuracy gains from step-by-step TableQA methods should be reported with the model scale stated, since the same method can show a double-digit gain and a double-digit loss across scales."],"supporting_citations":[{"why":"Introduces chain-of-thought prompting, the step-by-step paradigm the paper adapts to tables and tests across model scales.","marker":"Wei et al., 2023"},{"why":"Self-augmentation, the two-step baseline that Table-Logic is compared against on the same models and datasets.","marker":"Sui et al., 2024"},{"why":"Chain-of-table, a prior table-specific reasoning-chain method this work positions its own decomposition against.","marker":"Wang et al., 2024"},{"why":"Provides HybridQA, one of the two Wikipedia hybrid table/text datasets used for the headline results.","marker":"Chen et al., 2021"},{"why":"Provides WikiTableQuestions, the pure-table dataset used in the accuracy comparisons.","marker":"Pasupat and Liang, 2015"},{"why":"Provides TAT-QA, the financial hybrid dataset used to show the contrast is smaller on financial tables.","marker":"Zhu et al., 2021"},{"why":"Qwen technical report; source of the Qwen-1.5 72B and 7B models used as bigger/smaller pairs.","marker":"Bai et al., 2023"},{"why":"G-Eval, the GPT-4-based evaluation approach the authors adopt for judging answer correctness.","marker":"Liu et al., 2023"}],"fun_headline_variants":["Step-by-step prompts: big models up, small models down","Scale decides: same prompts boost 70B, hurt 7B","TableQA: step-by-step reasoning splits models by size","Prompt flow: +7.8% on 70B, -11% on 7B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire contrast rests on accuracy numbers produced by a single GPT-4 yes/no judge, and some of the headline differences are as small as 0.1 percentage points, so if the judge is noisy or biased the reported sign flip could be overstated.","fun_headline_variants_meta":{"raw":{"variants":["Step-by-step prompts: big models up, small models down","Scale decides: same prompts boost 70B, hurt 7B","TableQA: step-by-step reasoning splits models by size","Prompt flow: +7.8% on 70B, -11% on 7B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1815,"prompt_tokens":899,"completion_tokens":916,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":836}},"tokens_in":515,"tokens_out":916,"duration_ms":6874,"temperature":1.0,"reasoning_tokens":836,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:38:06.317177+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Llama-3-70B and Llama-2-7B on WikiTableQuestions under vanilla and Table-Logic prompts, scoring with a deterministic exact-match metric across multiple independent runs; if the sign flip between the two models disappears or reverses, the claimed contrast depends on the GPT-4 judge rather than on the reasoning method.","supporting_citations":[],"review_version":1}