{"id":"819c15b1-56ec-487c-94f4-6067c51c1c0c","arxiv_id":"2412.12386","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"POS decomposes table questions into atomic natural-language steps, executes each as SQL, and shows the intermediate tables, beating prior methods on explanation-quality tests while staying competitive on accuracy and cutting LLM calls.","lead":"Plan-of-SQLs is a Table QA method that splits each question into small natural-language steps, turns each step into an SQL command, and runs the commands in sequence so every intermediate table is visible. The paper reports that these step-by-step explanations help people and LLM judges verify answers better than prior methods, while keeping accuracy similar and using fewer LLM calls.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% faithfulness/comprehensiveness guarantee is contradicted by POS's own fallback path and FeTaQA extension; the central interpretability claim needs scoping before it can stand as stated.","rationale":"The reader's weakest-assumption field points to planning ability, and the paper's own Table 12 confirms planning errors dominate. That is a genuine performance limitation, but it does not undermine the paper's main interpretability thesis: even a wrong plan, executed through SQL, yields an explanation that is faithful to what the model actually did. The more load-bearing problem is the absolute-faithfulness guarantee, because it is the theoretical reason POS explanations should beat self-explanation and baseline explanations. The fallback path and the FeTaQA extension break that guarantee for a non-negligible fraction of samples, and the paper itself admits this in Section 5 and Appendix C. The issue is not a disagreement with consensus; it is an internal inconsistency between the guarantee stated in Sections 2.3 and 4.1.3 and the method as actually implemented and evaluated. This is addressable by scoping the claim, which is why I keep the reader's CONDITIONAL verdict rather than rejecting the paper. I agree partially with the reader: their rationale already mentions the fallback contradiction, though their formal weakest-assumption field emphasizes planning.","tokens_in":34820,"tokens_out":7446,"duration_ms":69038,"concrete_test":"Run POS from the released repository on the 2,024 TabFact and 4,344 WikiTQ test sets with the exact prompts in Appendix H, logging the fallback flag and the user-facing explanation artifact for every sample. For each fallback sample, determine whether the explanation contains a complete chain of executed SQL operations that determines the final answer. If any fallback sample lacks such a chain, the '100% faithfulness/comprehensiveness' guarantee is false as stated; the paper should explicitly scope the guarantee to non-fallback samples and exclude or separately report those samples in the interpretability studies. Additionally, recompute Table 10 after discarding fallback samples; if POS's margin over self-explanation narrows materially, the absolute-faithfulness claim is carrying the empirical result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central interpretability claim rests on an absolute-faithfulness argument that the paper itself contradicts. Section 2.3 and Section 4.1.3 state that POS 'guarantees 100% faithfulness and comprehensiveness' because explanations are grounded in actual executed SQL operations. Section 5, however, reports a fallback path: when SQL execution fails, an LLM answers end-to-end, triggered in 3.16% of TabFact and 13.58% of WikiTQ samples, and the accuracy numbers in Table 6 include this fallback. For those samples, no executed SQL chain produced the answer, so the explanation cannot be grounded in executed SQL operations. Appendix C further concedes that for FeTaQA, 'the final answer is generated through a black-box operation of LLMs—making POS explanations no longer 100% faithful or comprehensive.' Yet Section 4.4 presents FeTaQA results as part of POS, and the abstract generalizes the transparency guarantee. This is an internal inconsistency in a load-bearing subclaim: the absolute-faithfulness argument is used to justify POS's superiority over post-hoc self-explanation (Section 4.1.3, Table 10). If the guarantee holds only on a subset, the interpretability advantage needs rescoping and re-benchmarking. The planning bottleneck (Table 12) is real but does not by itself threaten the interpretability claim; the broken faithfulness guarantee does.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Plan-of-SQLs (POS), a Table QA method that decomposes a question into atomic natural-language steps, translates each step into an executable SQL command, and executes the commands sequentially so that intermediate tables and the final answer are produced programmatically. The authors evaluate explanation quality through preference ranking, forward simulation, and prediction verification with both human and LLM judges, and evaluate QA accuracy and efficiency on TabFact, WikiTQ, and FeTaQA. The central claims are that POS produces the highest-quality explanations among compared methods, maintains competitive QA accuracy with far fewer LLM calls and database queries, scales robustly to large tables, and that LLM judges agree strongly with human judges in explanation evaluation.","tokens_in":35083,"tokens_out":4642,"duration_ms":44200,"significance":"If the claims hold, POS is a genuinely useful contribution: it is a simple, reproducible, program-grounded Table QA design whose explanations can be inspected cell-by-cell, it comes with released code and data, and it is evaluated with behavioral XAI tasks (forward simulation and verification) rather than subjective ratings alone. The efficiency result (4 LLM calls vs. up to 100 for DATER) and the ablation isolating SQL execution as the main driver of interpretability are valuable and clearly presented. The main reservations are that the absolute faithfulness guarantee is contradicted by the paper's own fallback path and FeTaQA extension, and that some interpretability evidence rests on prompt asymmetries and small human samples without confidence intervals.","major_comments":[{"comment":"The paper's absolute-faithfulness claim is load-bearing and internally inconsistent. Section 2.3 states that POS 'guarantees 100% faithfulness and comprehensiveness' because explanations represent reasoning through executed SQL operations, and Section 4.1.3 uses this 'absolute faithfulness' to explain why POS beats post-hoc self-explanation. However, Section 5 reports a fallback path in which, if SQL execution fails, an LLM answers end-to-end, and this path is triggered in 3.16% of TabFact and 13.58% of WikiTQ samples; Table 6 reports accuracy including this fallback. Appendix C further states that for FeTaQA the final answer 'is generated through a black-box operation of LLMs—making POS explanations no longer 100% faithful or comprehensive,' yet Section 4.4 presents FeTaQA results as part of POS. The guarantee therefore does not hold for the full reported system. The authors should rescope the faithfulness/comprehensiveness claim to the executed-SQL subset, report interpretability metrics separately for fallback and non-fallback samples, and soften the abstract and Section 2.3 wording accordingly.","section":"Section 5 and Appendix C vs. Section 2.3 and Section 4.1.3"},{"comment":"The LLM-as-judge preference-ranking prompt is not symmetric across methods, so Table 1(a) is confounded. In Appendix G.3, the POS description lists three components and explicitly advertises attribution maps and cell highlighting, while the Text2SQL description says only that the generated SQL command serves as the explanation, and DATER receives a shorter two-item description. Since the judge is asked to rank on 'clarity, coherence, and helpfulness,' the richer textual description can itself produce the observed POS advantage (average ranks 1.01–1.55). The authors should rerun the preference ranking with equal-length, same-structure method descriptions, or supplement it with human preference rankings; otherwise the 'highest-quality explanations' claim rests on a prompt artifact.","section":"Appendix G.3 and Table 1(a)"},{"comment":"The human forward-simulation result is the strongest direct evidence for the interpretability claim, but it is reported without confidence intervals, significance tests, or participant-level variation. With 32 volunteer CS students and roughly 200 responses per method, the reported 93.00% for POS versus 83.68–86.50% for baselines could still reflect sampling variability, and the abstract's claim that POS 'markedly improves' users' ability to simulate decisions needs statistical support. A permutation test or clustered bootstrap over participants is needed. In addition, Table 4 reports 'Human Samples' of 153, 172, 177, and 171 per method, which does not match the stated 800 total responses, so the response-count accounting should be clarified.","section":"Section 4.1.1 and Table 1(b)"}],"minor_comments":[{"comment":"In the Text-to-SQL paragraph of Appendix A, the sentence 'Therefore, the we recruited Computer Science students (who have SQL expertise) for our human study' contains a duplicated article and should read 'Therefore, we recruited...'.","section":"Appendix A"},{"comment":"Table 10 reports a preference value of 5.00 for 'Self-explanation' while the four main methods are ranked on a 1-to-4 scale; the scale and the number of alternatives used for this table should be defined explicitly.","section":"Table 10 and Section 4.1.3"},{"comment":"The claim that every table-based question can be decomposed into atomic SQL steps because 'SQL is relationally complete' is too strong for the implemented system, which operates on a single local table and cannot access external or world knowledge, as the paper itself concedes later in Appendix C. A qualifying sentence would remove the apparent contradiction.","section":"Section 5"},{"comment":"The HTML snippet for cell highlighting uses 'background-color:90EE90' without a leading '#'; the intended value is presumably '#90EE90'.","section":"Appendix G.3"},{"comment":"For the table-size analysis, the paper compares against degradation numbers 'from Wang et al. (2024)' but only cites the source in text; please provide the exact table or numbers in the appendix so the comparison is verifiable.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well organized and the method is reproducible, but the central interpretability claim is currently overstated: the 100% faithfulness guarantee is contradicted by the fallback path and the FeTaQA extension, and the LLM-preference evidence has a prompt-asymmetry confound. These issues are fixable within the manuscript's scope, but they need to be addressed before the paper can be accepted. I would support acceptance after the authors rescope the claims, rerun or supplement the preference study with symmetric descriptions, and add uncertainty measures to the human evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plan-of-SQLs is a real method with a real result. Decomposing Table QA into atomic natural-language steps, executing each as SQL, and showing attribution maps lets people simulate the model's answer better than existing explanation methods, and it does so with far fewer LLM calls. The human forward-simulation numbers (93% vs. 84-86.5%) and the efficiency numbers (4 calls vs. 25-100) are the strongest parts. The method is reproducible, code and data are public, and the ablation showing SQL execution as the main driver of both accuracy and interpretability is informative.\n\nThe soft spots are real but mostly addressable. The biggest issue is the \"100% faithfulness and comprehensiveness\" claim in Sections 2.3 and 4.1.3. The paper itself contradicts it: Section 5 describes a fallback where SQL execution failure triggers an end-to-end LLM answer, on 3.16% of TabFact and 13.58% of WikiTQ samples, and the reported accuracy includes those samples. For those cases no executed SQL chain produced the answer, so explanations cannot be grounded in executed SQL. Appendix C likewise concedes that FeTaQA's final answer comes from a black-box LLM, making POS explanations there not 100% faithful. This matters because the absolute-faithfulness argument is load-bearing in the comparison against post-hoc self-explanation. The claim should be scoped to the SQL-executed subset, and the interpretability comparison should be re-run or re-reported on that basis.\n\nOther soft spots: the LLM preference-ranking prompts give each method a different amount of descriptive detail, with POS receiving a long and flattering description, so those rankings are confounded. The human study is 32 CS students with roughly 200 responses per method and no confidence intervals; enough for a signal, not for precision. FeTaQA is compared only against LLM-only baselines, so \"competitive\" there is weaker than on the other benchmarks. The planning bottleneck (70-95% of errors) is real but does not threaten the interpretability claim; it is a limitation of the overall system, not of the explanation method.\n\nThis paper deserves a serious referee. The method is novel enough, the core evidence is strong enough, and the flaws are fixable. The revision needs to rescope the faithfulness guarantee and clean up the judge-prompt asymmetry. If those are handled, this becomes a solid contribution to the interpretability-for-Table-QA literature.","headline":"A genuinely useful interpretable Table QA method with strong human-simulation evidence and a real efficiency win, but the paper overclaims a 100% faithfulness guarantee that its own fallback path contradicts.","tokens_in":35631,"tokens_out":1885,"would_cite":true,"duration_ms":18229,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Plan-of-SQLs makes LLM table question answering interpretable by decomposing each question into atomic natural-language steps, each translated into an executable SQL command, so every intermediate result is transparent.","keywords":["Table Question Answering","Interpretability","SQL","Large Language Models","Plan-of-SQLs","Explainable AI","Forward simulation","Attribution maps"],"falsifier":"Run the paper's own ablation on a fresh, larger set of non-expert users: keep the natural-language plan but replace SQL execution with direct black-box LLM table transformations, then measure verification accuracy. The paper's claim that SQL grounding is the key interpretability component predicts a clear drop; if users verify just as well without executed SQL, the central claim would fail.","tokens_in":34642,"feed_emoji":"📊","tokens_out":5925,"duration_ms":51664,"temperature":0.7,"pith_summary":"Plan-of-SQLs (POS) is a Table Question Answering method built around a simple idea: instead of asking an LLM to answer a question about a table in one opaque step, decompose the question into a short sequence of atomic natural-language steps, translate each step into one executable SQL command, and run them in order. Every intermediate table is therefore a visible artifact, and the final answer is produced by deterministic SQL rather than by hidden LLM reasoning. The paper claims that these step-by-step, SQL-grounded explanations help users simulate and verify the model's decisions better than explanations from Text-to-SQL, DATER, or CoTable, while QA accuracy stays within one or two points of the best hybrid methods. It also reports large efficiency gains (four LLM calls and two database queries per question, versus up to 100 calls and 50 queries for baselines) and stable accuracy on large tables, where baselines degrade. The main caveat the paper itself identifies is that the planner's step sequence is the bottleneck: planning errors account for 70.7–95.4% of POS failures.","feed_headline":"Plan-of-SQLs turns LLM table reasoning into auditable SQL steps","feed_subtitle":"Users verify POS answers better than with prior methods, at up to 25x fewer queries.","key_machinery":"The load-bearing mechanism is the atomic step, defined as a minimal natural-language table operation restricted to at most one condition and one variable (for example, \"Select rows where opponents = 0\"), which the Step-to-SQL module translates into a single SQL statement (SQL being the standard language for querying relational tables). The NL Atomic Planner produces the step sequence; each SQL executes on the current intermediate table, and an attribution-map generator marks the rows, columns, and cells used at every step, producing a chain of highlighted intermediate tables. This chain is the explanation: it shows which data contributed to the answer, when, and through which operation.","core_discovery":"The paper's central claim is that interpretability in Table QA can be achieved by making the reasoning process literally identical to the computation: each atomic step is a natural-language operation with at most one condition and one variable, converted into a single SQL statement and executed on the current table. Because the explanation is the executed trace, POS explanations are faithful and comprehensive by construction, with no separate reasoning to explain. On TabFact, WikiTQ, and FeTaQA, POS achieves accuracy competitive with existing methods (within 1–2 points), while its explanations rank first in preference, simulation, and verification studies with both human and LLM judges, and it requires up to 25x fewer LLM calls and database queries than baselines. The paper is careful to note that this full transparency applies to the SQL-executed steps; for free-form FeTaQA answers, the final narrative generation is delegated to an LLM, so those explanations are not claimed to be 100% faithful.","pith_inferences":["Because every reasoning step is an executable SQL statement, one could build an automated verifier that re-executes each step and checks the final answer against the plan, turning explanation quality from a subjective rating into a formal property; the paper sketches error detection but does not build this.","The paper measures explanation quality on semi-structured benchmark tables; an untested corollary is that POS's atomicity should make it relatively robust to messy real-world tables once a normalization preprocessor is added, which the authors suggest as future work.","The 70.7–95.4% planning-error share implies that improvements in LLM planning ability should translate almost directly into POS accuracy gains, so POS is a planner-limited rather than an executor-limited architecture; this is an interpretation consistent with the paper's own error analysis."],"forward_implications":["If POS is right, explanation quality in Table QA does not have to be traded against accuracy: users get both within one or two accuracy points of state-of-the-art hybrid methods.","The up-to-25x reduction in LLM calls and database queries means POS is substantially cheaper to deploy at scale than DATER, Binder, or CoTable.","Because POS keeps accuracy stable across table sizes while baselines drop by 30–50 points on large tables, it is a stronger candidate for real-world tables that exceed LLM context limits.","The high agreement between LLM and human judges (up to 90.59% in forward simulation) suggests that LLM judges can be used as proxies in early-phase explanation evaluation.","For free-form answers, POS's intermediate SQL steps still improve BLEU and ROUGE over end-to-end and few-shot QA, indicating the decomposition helps even when a final LLM writing step is required."],"supporting_citations":[{"why":"Provides the Text-to-SQL baseline that POS is compared against for explanation quality and accuracy.","marker":"Rajkumar et al. (2022)"},{"why":"Provides the DATER baseline; POS must beat its subtable-based explanations and hybrid accuracy.","marker":"Ye et al. (2023)"},{"why":"Provides the Chain-of-Table (CoTable) baseline and is the source of the cleaned TabFact data and efficiency benchmark.","marker":"Wang et al. (2024)"},{"why":"Provides the self-explanation baseline that POS outperforms, motivating the faithfulness argument.","marker":"Madsen et al. (2024)"},{"why":"Supplies the TabFact dataset used for fact-verification benchmarks.","marker":"Chen et al. (2020)"},{"why":"Supplies the WikiTQ dataset and evaluation scripts used for denotation accuracy.","marker":"Pasupat & Liang (2015)"},{"why":"Supplies the FeTaQA dataset used to test free-form Table QA.","marker":"Nan et al. (2022)"},{"why":"Provides the forward-simulation and verification evaluation framework the paper follows.","marker":"Doshi-Velez & Kim (2017)"}],"fun_headline_variants":["POS: table QA with auditable SQL steps, up to 25x fewer queries","Plan-of-SQLs: transparent table QA via executable SQL traces","Table QA you can verify: POS's SQL steps beat opaque LLMs","Auditable reasoning: Plan-of-SQLs cuts table QA queries by 25x","Step-by-step SQL makes LLM table answers trustworthy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method assumes the LLM planner will write a correct sequence of atomic steps; the paper's own error analysis shows planning mistakes cause 70.7–95.4% of failures, so if the planner is weak, both accuracy and explanation quality collapse.","fun_headline_variants_meta":{"raw":{"variants":["POS: table QA with auditable SQL steps, up to 25x fewer queries","Plan-of-SQLs: transparent table QA via executable SQL traces","Table QA you can verify: POS's SQL steps beat opaque LLMs","Auditable reasoning: Plan-of-SQLs cuts table QA queries by 25x","Step-by-step SQL makes LLM table answers trustworthy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000648,"raw_usage":{"total_tokens":2988,"prompt_tokens":973,"completion_tokens":2015,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":1918}},"tokens_in":589,"tokens_out":2015,"duration_ms":13661,"temperature":1.0,"reasoning_tokens":1918,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:08:16.970104+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's own ablation on a fresh, larger set of non-expert users: keep the natural-language plan but replace SQL execution with direct black-box LLM table transformations, then measure verification accuracy. The paper's claim that SQL grounding is the key interpretability component predicts a clear drop; if users verify just as well without executed SQL, the central claim would fail.","supporting_citations":[],"review_version":1}