{"id":"2df68bcb-7435-4e6e-a751-1b28a5389779","arxiv_id":"2607.11207","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"A Python-preprocessing + CoT-SQL pipeline lets LLMs reason over large, inconsistently formatted tables more accurately than prior prompting baselines.","lead":"ProgramTab is a five-step LLM pipeline that first ranks table rows by embedding similarity, extracts relevant columns, rewrites messy web-table cells into clean typed columns via generated Python, then issues CoT SQL and answers from the result. It reports state-of-the-art accuracy among prompting methods on WikiTQ and TabFact while staying under context limits on large tables.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Top-K=10 GTE exemplars (only) drive column/code/SQL generation while full table is executed later; 86% answer coverage is an untested proxy for whether those exemplars suffice, especially on large/aggregation tables.","rationale":"Reader correctly flags the frozen K=10 retrieval as the weakest assumption supporting the central claim. Close reading of §§3.1–3.4 and Figs. 2–5 confirms that retrieval supplies only prompt exemplars; execution is full-table, so the 86% proxy is even weaker than it first appears (format diversity and multi-row needs are unmeasured). Ablations already isolate the value of data-redefinition and CoT-SQL, but both presuppose adequate exemplars. No internal contradiction or inflated numbers appear; the engineering is clean and the large-table results are consistent with the pipeline. Hence the CONDITIONAL verdict (pending code release and a thorough retrieval analysis) stands; the proposed stratification/random-row test would settle the residual risk without requiring new models.","tokens_in":16594,"tokens_out":546,"duration_ms":41794,"concrete_test":"On WikiTQ (and its >4000-token large-table subset) stratify by whether gold answer-containing rows lie inside the top-10 GTE ranks; report ProgramTab accuracy on covered vs. uncovered slices. Separately re-run the identical pipeline with K=10 randomly sampled rows; if overall accuracy falls >5 points (or >10 on large tables) the retrieval assumption is load-bearing and the robustness claim requires qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The length-robust outperformance claim (Tables 1-2, 4) rests on LLMs producing correct column choices, Python normalizers (Fig. 4), and CoT-SQL (Fig. 5) from headers + only the top-10 GTE-base rows. Full T (after column filter) is used solely at execution time. Appendix A.4's 86% \"answer in top-10\" therefore does not guarantee that format variants, rare filter values, or multi-row statistics needed for correct code/SQL are exemplified. When they are not (plausible for large tables or aggregation questions), silent failures in normalization or predicates can occur; no ablation swaps retrieved rows for random/oracle/bottom-K rows, nor stratifies final accuracy by coverage. Thus the large-table gains and the \"outperforms all LLM baselines\" headline inherit an unquantified retrieval assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces ProgramTab, a five-stage in-context learning pipeline for LLM table reasoning: (1) embedding-based top-K row retrieval (GTE-base, K=10), (2) LLM column extraction, (3) LLM-generated Python code that standardizes formats, casts types, and optionally derives new columns, (4) CoT-decomposed text-to-SQL executed on the cleaned table, and (5) final answer generation from the SQL result. It targets two known failure modes—context-length limits on large tables and the structural inconsistency of web tables that makes pure SQL brittle. Experiments on WikiTQ, TabFact and (appendix) HiTab with GPT-3.5-Turbo, Llama-3.1-70B-Instruct and GPT-4o-mini report state-of-the-art numbers among prompting methods (e.g., 76.0 % denotation accuracy on WikiTQ and 89.6 % on TabFact with GPT-4o-mini), strong large-table and token-cutoff results, ablations isolating the Python-redefinition and CoT-SQL stages, efficiency counts, and an error taxonomy.","tokens_in":16882,"tokens_out":1241,"duration_ms":20687,"significance":"If the empirical claims hold, ProgramTab supplies a practical, training-free recipe that simultaneously mitigates length constraints and data-quality problems that currently limit LLM table reasoners. The explicit separation of Python normalization from SQL extraction is a clean engineering contribution; the efficiency analysis (only four LLM calls, ~4 s/sample) and the large-table stress tests (Table 4) are concrete strengths that go beyond many concurrent prompting papers. The work is therefore of clear interest to the table-reasoning and LLM-tool-use communities, provided the retrieval assumption that underpins the length-robustness claims is more thoroughly validated.","major_comments":[{"comment":"§3.1 and Appendix A.4 state that top-K=10 GTE-base rows are used as the sole exemplars for column selection, Python redefinition and CoT-SQL generation, while the full (column-filtered) table is used only at execution time. The sole supporting statistic is that the gold answer appears among those 10 rows for 86 % of WikiTQ examples. No ablation replaces top-K with random, bottom-K or oracle rows, nor is final accuracy stratified by whether the answer (or the format variants / rare filter values needed for correct code) is present in the exemplars. Because the large-table gains (Table 4) and the headline “outperforms all LLM-based baselines” rest on this untested proxy, the central length-robustness claim remains incompletely supported.","section":"§3.1, Appendix A.4, Table 4"},{"comment":"Table 3 shows a 10.9-point drop on WikiTQ when data redefinition is removed, confirming that the Python stage is load-bearing. Yet the paper never reports how often the generated Python actually alters column types or extracts new columns, nor whether those alterations are correct on the full table versus the top-K exemplars. Without such diagnostics it is hard to know whether the observed gains come from genuine normalization or from incidental side-effects of the few-shot prompt.","section":"§5.1, Table 3, Figure 4"},{"comment":"Implementation details (§4.3) state that after five failed retries on syntax errors the pipeline simply aborts. The fraction of such hard failures, and whether they are counted as incorrect answers, is never reported. Given that code and SQL errors dominate the manual error analysis (Figure 7), the missing failure-rate statistic leaves open the possibility that a non-negligible portion of the reported accuracy gap versus baselines is an artifact of the retry policy rather than of better reasoning.","section":"§4.3, Figure 7"}],"minor_comments":[{"comment":"Abstract and §1 use the phrase “momentous contents extraction”; “key” or “salient” would be clearer.","section":"Abstract, §1"},{"comment":"Figure 6 prompt contains the typo “venus” for “venue”; the same figure also mixes “SQL Answer” and “A:” labels inconsistently.","section":"Figure 6"},{"comment":"Table 1 and Table 2 mix GPT-4, Codex and GPT-4o-mini baselines; a short note clarifying that each ProgramTab row is compared only against methods that used the same (or weaker) backbone would avoid any appearance of unfair comparison.","section":"Tables 1–2"},{"comment":"Appendix A.5 HiTab numbers are given only for GPT-4o-mini; reporting the same three backbones used in the main tables would strengthen the hierarchical-table claim.","section":"Appendix A.5"},{"comment":"The hyper-parameter table (temperature 0.6, top_p=1, K=10, five retries) is buried in Appendix A.2; moving the key values into the main experimental section would improve reproducibility.","section":"§4.3, Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The differentiation from NormTab (direct table rewriting vs. code generation) is stated clearly and appears genuine; I see no novelty or citation-pattern concerns. The main risk is that the un-ablated retrieval assumption may not survive reviewer scrutiny, so a major-revision decision that forces the authors to quantify it is the cleanest path."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is that ProgramTab is a practical five-stage prompting recipe (GTE row retrieval, LLM column pick, Python redefinition, CoT SQL, answer gen) that cleanly shows why normalizing messy web-table formats with code before SQL helps, and it posts the best numbers among the pure-prompting baselines on WikiTQ and TabFact across three backbones.\n\nWhat is actually new is the explicit separation of concerns: use an embedding model to surface only the top-10 rows as exemplars for the LLM, let the model write a short pandas function that unifies formats and types (and optionally adds derived columns), then generate multi-step SQL on the cleaned schema. The ablation in Table 3 is the strongest evidence—they drop data redefinition and lose 10.9 points on WikiTQ—so the Python stage is doing real work rather than being decorative. Large-table and token-cutoff experiments (Tables 4-5) also hold up better than Binder/Dater/TabSQLify, and the efficiency count (only four LLM calls, greedy) is honest. Error analysis and the HiTab add-on are useful extras. Citations cover the right prior art (NormTab, Dater, Chain-of-Table, etc.) without obvious gaps.\n\nThe soft spot is real but proportionate: everything after row retrieval (column choice, code, SQL) is conditioned only on those top-10 GTE rows plus headers. Appendix A.4’s 86 % “answer appears in top-10” is a weak proxy for whether the exemplars contain the format variants or rare values needed to write correct normalizers and predicates. No swap-in of random/oracle/bottom-K rows, no stratification by coverage, and K is frozen at 10. That assumption quietly underpins the length-robustness claim. Missing code release is another practical gap. Novelty is incremental—an engineering synthesis, not a conceptual leap—but the paper never pretends otherwise.\n\nThis is for people building table-QA systems who need a length-robust, low-sample recipe they can actually run. The math is none, the data are standard public sets, the numbers look solid. I would send it to peer review; a referee can demand the retrieval ablation and code. Worth reading if you work in the area; not essential if you don’t.","headline":"Clean engineering pipeline that puts Python normalization before CoT-SQL; consistent gains and large-table robustness, but the fixed top-K=10 retrieval that drives code/SQL generation is under-tested.","tokens_in":17473,"tokens_out":585,"would_cite":true,"duration_ms":15717,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"ProgramTab teaches LLMs to clean messy web tables with Python before extracting answers via SQL, outperforming other prompting methods on table QA and fact verification.","keywords":["table reasoning","large language models","in-context learning","Python code generation","text-to-SQL","table preprocessing","WikiTQ","TabFact"],"falsifier":"On a held-out set of large tables where the gold answer rows rank outside the top-10 by the same embedding model, measure whether ProgramTab accuracy collapses relative to an oracle that is given the true relevant rows.","tokens_in":17510,"feed_emoji":"📊","tokens_out":787,"duration_ms":15039,"temperature":0.7,"pith_summary":"Large language models struggle with table reasoning when tables are large or inconsistently formatted: long inputs hit context limits, and raw web tables often lack the structure and types needed for reliable SQL. ProgramTab addresses this by guiding an LLM, via few-shot prompts, through five steps: retrieve the most relevant rows with embeddings, select useful columns, generate Python code that standardizes formats and types and optionally adds derived columns, produce multi-step SQL with chain-of-thought, then read the SQL result to answer the question. On WikiTQ and TabFact the method beats prior LLM baselines under the same models, and it remains strong even when tables exceed typical context windows. The practical payoff is a length-agnostic pipeline that turns messy tables into clean, queryable data without extra training.","feed_headline":"Python cleaning plus SQL lifts LLM table accuracy","feed_subtitle":"ProgramTab normalizes messy web tables first, then queries them, beating prior prompting methods even on long tables.","key_machinery":"ProgramTab: a five-stage in-context pipeline (row retrieval by embedding, column extraction, Python data-redefinition code, CoT SQL generation, answer synthesis) that first normalizes table cells then extracts the answer.","core_discovery":"By inserting an explicit Python preprocessing stage before SQL generation, and by decomposing questions into sub-queries, LLMs can handle large, inconsistently structured tables far more reliably than methods that feed whole tables or generate SQL alone; the resulting ProgramTab pipeline outperforms all compared LLM-based baselines on WikiTQ and TabFact while using only four LLM calls per example.","pith_inferences":["If row-retrieval coverage is the main failure mode, replacing the frozen embedder with a learned or iterative retriever should lift the remaining errors more than stronger code models alone.","The same Python-first normalization idea could be reused for semi-structured documents (HTML lists, CSV dumps) that currently break pure SQL agents.","Error analysis suggests code and SQL generation quality scale with model strength; weaker open models may need self-repair loops the paper currently limits to five retries."],"forward_implications":["Table-reasoning systems can drop full-table context and still improve accuracy by first emitting executable Python normalizers.","Existing text-to-SQL pipelines become more robust on web tables once a short code-generation step unifies formats and types.","Latency and token cost stay low because only four greedy LLM samples are required per question.","The same normalize-then-query pattern extends immediately to hierarchical tables after a lightweight header-flattening pass."],"fun_headline_variants":["Python prep before SQL lifts LLM table reasoning","ProgramTab cleans tables in Python then queries via SQL","Code stage plus sub-queries help LLMs on large messy tables","Python cleaning then SQL extraction beats pure prompting","ProgramTab: four LLM calls outperform baselines on WikiTQ"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The top-10 rows retrieved by a frozen embedding model, plus the columns chosen in one LLM pass, still contain the information needed to answer most questions.","fun_headline_variants_meta":{"raw":{"variants":["Python prep before SQL lifts LLM table reasoning","ProgramTab cleans tables in Python then queries via SQL","Code stage plus sub-queries help LLMs on large messy tables","Python cleaning then SQL extraction beats pure prompting","ProgramTab: four LLM calls outperform baselines on WikiTQ"]},"model":"grok-4.5","effort":"low","cost_usd":0.004152,"raw_usage":{"total_tokens":1170,"prompt_tokens":717,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":41520000,"prompt_tokens_details":{"text_tokens":717,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":394,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":717,"tokens_out":59,"duration_ms":3539,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T06:16:13.373648+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out set of large tables where the gold answer rows rank outside the top-10 by the same embedding model, measure whether ProgramTab accuracy collapses relative to an oracle that is given the true relevant rows.","supporting_citations":[],"review_version":1}