{"id":"61b65b48-7b04-482c-9991-387fe5c1ae70","arxiv_id":"2506.23463","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ATF prunes table columns and rows with LLM scoring plus retrieval, cutting cells by about 70% and improving out-of-domain TableQA accuracy, while hurting in-domain QA and fact verification.","lead":"This paper proposes ATF, a preprocessing pipeline that uses an LLM to score and prune table columns and rows before feeding the table to question-answering models. Tests on out-of-domain TableQA datasets show the method cuts table size by about 70% and often improves answer accuracy, though it hurts in-domain QA and fact verification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported raw TAPAS baseline on out-of-domain Open-WikiTable (EM 0.599) exceeds its in-domain WikiTableQuestions score (EM 0.296) by 2x, which is suspicious for a dataset the paper itself describes as requiring more complex reasoning.","rationale":"The paper is a genuinely useful modular preprocessing idea: column-level LLM scoring plus sparse-dense row ranking is a reasonable, testable recipe, and the authors are transparent that ATF hurts in-domain QA and Table Fact Verification (Tables 1 and 3). The ablation study (Table 5) shows both stages contribute, and the case studies illustrate the intended behavior. These are real strengths. However, the central claim is an empirical one about OOD gains, and the reported raw-base-number inversion is the most load-bearing weak point I can find. The reader's identified assumption (fixed alpha=0.4 dropping 60% of rows) is a mechanism-level fragility that could limit generalizability, but it does not directly invalidate the measured improvement on the two OOD datasets; in fact, the reported results could still hold even for questions where the gold row is not in the top 40%, as long as the retained rows contain enough evidence. By contrast, if the raw baseline itself is not reproducible, then the measured improvement is not trustworthy at all. I therefore disagree with the reader's choice of the single weakest assumption, though I partially agree with their overall skepticism about evaluation validity (they also note missing error bars and hyperparameter tuning). My concrete test—re-running the released checkpoint on the full OOD test sets with the paper's normalization—would settle whether the baseline is valid. If it reproduces, then the remaining concerns (no error bars, tuned alpha/K, hard-to-verify code) still justify a conditional rather than an outright accept, but not a rejection. If it does not reproduce, the empirical claim is unsupported and the paper would need re-evaluation. Since my concern does not by itself change the reader's CONDITIONAL verdict—it strengthens one of the conditions—I recommend UNCHANGED.","tokens_in":23648,"tokens_out":9914,"duration_ms":104834,"concrete_test":"Download the released TAPAS-base-wtq checkpoint (from the ATF repository or the original TAPAS repository) and run the paper's evaluation pipeline on the full Open-WikiTable test set (and, as a control, the WTQ test set) using the paper's exact answer-normalization and EM/F1 code. Record the raw and ATF-filtered EM/F1. Then check for n-gram overlap between Open-WikiTable questions/tables and the WTQ training split to rule out leakage. If the raw Open-WikiTable EM is close to 0.599 and WTQ EM is close to 0.296 with the same script, the baseline is reproduced and the central claim stands (modulo the usual lack of error bars). If either number shifts materially (e.g., OOD EM falls below the ID EM), the reported OOD improvement is an artifact of the comparison, not of ATF.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is purely empirical: ATF improves or maintains performance on OOD TableQA tasks. The evidence for this rests on the numbers in Tables 1 and 2. In Table 1, TAPAS-base achieves 0.296 EM on WikiTableQuestions (the in-domain dataset used for fine-tuning). In Table 2, the same (presumed) TAPAS-base achieves 0.599 EM on Open-WikiTable, an OOD dataset the authors call 'more complex reasoning, such as multi-row inference and latent column operations.' A model fine-tuned on WTQ should not nearly double its EM on a harder, unseen dataset; this inversion contradicts the paper's own framing of OOD difficulty. The discrepancy suggests one of the following: different checkpoints are used across tables, the OOD evaluation is run on a subset or with a different metric, the answer normalization differs, or the Open-WikiTable test set contains leakage (e.g., table/answer templates that make lexical lookup trivial). If the raw baseline is actually lower (e.g., ~0.3), then ATF's '+1.8%' on Open-WikiTable could become a large relative gain or a loss, and the AIT-QA improvement (+35.4% EM for TAPAS) is likewise uninterpretable. The overflow statistics in Table 7 do not resolve this: raw TAPAS overflows on only 22.2% of Open-WikiTable and 21.8% of AIT-QA examples, so 78% of the OOD examples are already inside the 512-token window; the reported large gains on top of that need a credible baseline before they can be attributed to filtering. Without an independent run of the released checkpoint on the full OOD test sets, the central empirical claim is not yet secured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ATF, a question-aware preprocessing framework that prunes table columns and rows before a downstream TableQA model (TAPAS/TAPEX) sees the table. Column pruning uses LLM-generated semantic descriptions, repeated LLM relevance scoring with variance adjustment, dense cosine similarity, and K-means clustering with an ensemble of cluster-selection criteria; row pruning uses a softmax-fused combination of TF-IDF, BM25, and Sentence-BERT scores with a fixed 40% row-retention ratio. The central empirical claim is that ATF reduces table cells by up to 70% while improving or maintaining EM/F1 on out-of-domain TableQA datasets (Open-WikiTable, AIT-QA) without retraining the QA model, while it slightly degrades in-domain WikiTableQuestions and Table Fact Verification. The paper includes ablations, token-overflow statistics, and several worked examples.","tokens_in":24090,"tokens_out":6842,"duration_ms":68121,"significance":"If the empirical claim holds, ATF is a useful, model-agnostic preprocessing lever: it is cheap, requires no downstream fine-tuning, and it is accompanied by released code and detailed prompts, which strengthens reproducibility. The claim is falsifiable and the paper is honest about limitations, reporting degradations on in-domain TableQA and on Table Fact Verification. However, the evidence is not yet conclusive. The raw OOD baselines in Table 2 are internally suspicious, no error bars or significance tests are provided, several hyperparameters appear to be tuned on the evaluation sets, and there is no comparison to existing filtering/retrieval methods. The paper's practical value is real if these issues are resolved, but the current experimental support is insufficient for the strength of the stated claims.","major_comments":[{"comment":"The raw TAPAS-base numbers are internally inconsistent with the paper's own difficulty framing. On the in-domain WikiTableQuestions test set, TAPAS-base achieves EM 0.296 (Table 1), but the same model reportedly achieves EM 0.599 on Open-WikiTable, a dataset the paper describes as requiring 'more complex reasoning, such as multi-row inference and latent column operations.' A model fine-tuned on WTQ should not nearly double its EM on an unseen, harder dataset. Please state explicitly whether exactly the same checkpoint, preprocessing, evaluation script, answer normalization, and test split are used in both tables; if different resources were used, the reported ATF gains (+1.8% on Open-WikiTable and +35.4% on AIT-QA) are not interpretable until the raw baseline is recomputed consistently.","section":"§4.4, Tables 1 and 2"},{"comment":"The overflow statistics in Table 7 do not support a truncation-based explanation of the gains: for Open-WikiTable, only 22.2% of raw TAPAS inputs exceed the 512-token limit, and for AIT-QA only 21.8%, so more than three quarters of the OOD examples are already inside the window. The reported benefits must therefore come from improved reasoning on in-window tables, yet all tables report single runs with no standard deviation, confidence intervals, or significance tests. The Open-WikiTable EM gain is only 0.011 absolute (0.599 vs. 0.610), which is well within plausible run-to-run noise for a TAPAS evaluation. Please add multiple seeds or bootstrap confidence intervals, and report per-dataset variance.","section":"§4.4, Table 7"},{"comment":"Several central hyperparameters appear to be selected with knowledge of the evaluation data. The row-fusion weights (0.4 TF-IDF, 0.3 BM25, 0.3 dense) are described as 'empirically chosen' without a separate validation split; the MCDM weights (0.4, 0.2, 0.2, 0.2), confidence weights (0.4, 0.4, 0.2), and the threshold tau = 0.7 are fixed constants; and K = 3 in K-means is selected using Elbow/Silhouette analyses whose data provenance is not specified. If these choices were tuned on Open-WikiTable and AIT-QA, then the 'out-of-domain' claim is not fully held-out. Please report a validation-based selection procedure or a sensitivity analysis, and clarify which datasets were used to set each hyperparameter.","section":"§3.9.3, §3.9.4, §4.7"},{"comment":"The fixed row-retention ratio alpha = 0.4 always discards 60% of rows before the QA model sees the table. For questions that require aggregation over many rows, or where the answer row is not lexically or semantically similar to the question, the filtered table cannot contain the evidence needed for the answer. This is a mechanism-level premise that is not validated separately for the OOD datasets. The paper does not report performance broken down by reasoning type (e.g., count, comparison, lookup), so it is unclear how often this failure mode occurs and whether it explains the in-domain degradation or the TFV degradation.","section":"§3.9.4, Eq. (19)"},{"comment":"The experimental comparison is limited to raw tables and a Top-K ablation; no existing filtering or table-compression method is used as a baseline, despite a detailed related-work discussion of MATE, DATER, Chain-of-Table, TableRAG, and ALTER. Given the paper's claim that ATF is a novel modular framework, at least one comparison to an existing preprocessing or retrieval-based subsetting method is needed to establish that the observed OOD gains are specific to ATF rather than a general property of any filtering front-end. If such a comparison is not feasible, the claims should be tempered to 'filtering can help' rather than 'ATF is superior.'","section":"§2.2, §4.4"}],"minor_comments":[{"comment":"There are numerous typographical errors, including 'fintuned' (§4.2), 'Essentail' (Table 11 header area), 'Anser' (Table 9), and the erroneous appearance of the column 'nominated for' in the AIT-QA cluster description of Table 11. A thorough proofread is needed.","section":"Throughout"},{"comment":"The 'Filtered Table (Output)' in the AIT-QA case study shows all six original columns, even though the 'Final Selected Columns' lists only four columns. This internal inconsistency in a worked example undermines confidence in the implementation; please correct the example or the pipeline description.","section":"Table 11"},{"comment":"The sentence 'This semantic abstraction serves as an intermediate representation that enhances [Wang et al., 2024] interpretability' is ungrammatical and appears to place a citation in the wrong position; the intended reference is unclear.","section":"§3.5.1"},{"comment":"The 7.5% relative drop for TAPAS-large on SEM-TAB-FACT test_a (0.751 to 0.695) is described in the text as 'slight,' but it is larger than the other reported drops; please use a more neutral characterization and report absolute percentage-point differences as well.","section":"Table 3"},{"comment":"In the ablation table, the 'w/o column filtering' row shows EM 0.606 for TAPAS, which is higher than the raw baseline EM 0.599 but the text says performance degrades when either component is removed. The comparison baseline is presumably the full ATF (0.610) rather than the raw model; please make this explicit in the text to avoid confusion.","section":"§4.5, Table 5"},{"comment":"The text mentions 'Figure 4 (Bottom)' when referring to the Silhouette Score plot, but the plot is labeled Figure 9; the cross-reference is incorrect.","section":"§4.6, Figure 9"},{"comment":"The meaning of 'Cell ↓' is not defined; state whether the reported percentages correspond to cell-level, token-level, or both, and specify the averaging method across tables.","section":"Table 1, Table 2"},{"comment":"The Limitations section appropriately acknowledges in-domain and TFV degradation and LLM-induced latency; however, the main concern about baseline credibility and hyperparameter tuning is not acknowledged there and should be addressed in the revision.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The raw-baseline discrepancy in Table 2 is the single most important issue to verify. If the Open-WikiTable and AIT-QA raw scores are not from the exact same TAPAS/TAPEX checkpoints and normalization pipeline as the WTQ scores, the paper's central claim is not supported. I recommend the editor ask the author to recompute all baselines consistently and to report confidence intervals. There is also a scope/fit question: the paper reads more like a workshop paper than a full journal contribution, but the idea is solid and the code release is a positive. A comparison to at least one existing filtering method would substantially raise the bar."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick read of the ATF paper. The pipeline is real and the reporting is unusually honest: they show in-domain degradation on WTQ and TabFact clearly, and they frame ATF as an OOD robustness tool rather than a universal boost. That framing, plus the modular plug-and-play design (LLM column descriptions, variance-adjusted scoring, k-means on 2D score vectors, cluster ensemble, hybrid sparse-dense row ranking), is the actual contribution. It is a sensible engineering combination, not a new theoretical idea, and it is close to ALTER and TableRAG but distinct enough, especially the clustering and the fine-tuned TAPAS/TAPEX compatibility.\n\nThe problem is the central empirical claim. TAPAS-base gets 0.296 EM on WTQ (its fine-tuning set) and 0.599 on Open-WikiTable, which the paper itself calls more complex. That inversion is not explained. It could be a different checkpoint, a different evaluation subset, answer normalization differences, or leakage; whatever it is, the reader cannot attribute ATF's gains to filtering until the baselines are credible. The overflow statistics show only 22.2% of Open-WikiTable exceeds TAPAS's 512-token limit, so the 'handles larger tables' story is secondary. On AIT-QA, the +35.4% EM for TAPAS is similarly uninterpretable without a trustworthy raw baseline.\n\nOther soft spots: all reported numbers are single runs with no error bars or significance tests. K=3, alpha=0.4, and the fusion weights were tuned on the evaluation benchmarks, not held out. The code repository exists but the paper doesn't include a reproducibility statement that would let a reader verify the numbers.\n\nThat said, the method is not a mirage. The ablations (Table 5, 6) show both stages matter and clustering beats top-k, and the limitations section is candid. If the baselines are fixed or properly re-run, this could be a useful preprocessing contribution for practitioners working with noisy large tables. As it stands, the headline OOD claim is not secured.\n\nMy recommendation: send it to review, but with a strong request for independent runs, error bars, and clarification of the Open-WikiTable baseline discrepancy. The paper deserves referee time because the framework is plausible, well-organized, and honest, but it needs a revision that addresses the empirical gap before acceptance. I would not cite it in my own work until that is done.","headline":"A practical table-filtering pipeline with honest caveats, but the headline OOD gains rest on baseline numbers that do not add up on their face.","tokens_in":24609,"tokens_out":2890,"would_cite":false,"duration_ms":25412,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ATF, a question-aware table-filtering pipeline, claims to cut table cells by up to 70% while improving or maintaining out-of-domain TableQA accuracy without retraining.","keywords":["Adaptive Table Filtering","table question answering","table pruning","LLM relevance scoring","sparse-dense retrieval","out-of-domain generalization","table fact verification","input compression"],"falsifier":"Run ATF on an out-of-domain TableQA set where every question requires aggregating over more than 40% of the rows (for example, asking for a sum or average over all rows in the table); if accuracy drops below the unfiltered baseline, the claim that ATF preserves out-of-domain QA accuracy fails because the fixed row-retention rule systematically removes needed rows.","tokens_in":23463,"feed_emoji":"📊","tokens_out":8458,"duration_ms":83120,"temperature":0.7,"pith_summary":"The paper introduces ATF, a preprocessing pipeline that decides which columns and rows of a table matter for a given question and prunes the rest before a question-answering model reads the table. It claims that this can cut table cells by up to 70% and, on out-of-domain TableQA benchmarks, improve or preserve accuracy without retraining or changing the downstream model. The benefit comes with a documented trade-off: on in-domain data and on table fact verification, where the whole table is needed, filtering hurts. A reader should care because a cheap, model-agnostic front-end that shrinks tables would let existing models handle larger and noisier real-world tables.","feed_headline":"Table filter cuts cells by 70% and lifts out-of-domain QA","feed_subtitle":"A preprocessing pass prunes columns and rows before the QA model runs, improving unseen table formats with no retraining.","key_machinery":"ATF's machinery is a five-stage, question-conditioned pruning pipeline. First, an LLM predicts the answer entity type. Second, an LLM names essential columns that are always kept. Third, each column gets a generated semantic description, then repeated LLM relevance scores (averaged and discounted by their standard deviation) plus a cosine-similarity score from sentence embeddings. Fourth, columns are clustered by their two scores with K-means, and the best cluster is chosen by majority vote of three selection criteria, with the top column from each other cluster and all essential columns added. Fifth, rows are serialized only through the selected columns, scored by TF-IDF, BM25, and dense similarity, softmax-normalized, fused with weights 0.4/0.3/0.3, and the top $\\lceil 0.4n \\rceil$ rows are kept. The final table is $T[R', C']$.","core_discovery":"The paper's central discovery is that a large share of TableQA can be answered from a small, question-relevant fragment of a table, and that fragment can be found before the QA model runs. ATF uses an LLM to describe and score columns, clusters columns by those scores, then scores rows against the question using sparse and dense retrieval, keeping the top 40% of rows. On out-of-domain benchmarks (Open-WikiTable and AIT-QA), applying this front-end to two fine-tuned table-parsing models improved exact match and F1 while cutting cells by 67-70%; on AIT-QA the small base model with ATF matched a larger unfiltered model on exact match. The same filtering consistently hurt in-domain WikiTableQuestions and table fact verification, which the paper attributes to those settings needing the full table context the model was trained on.","pith_inferences":["The in-domain degradation suggests a direct follow-up: fine-tune the downstream model on ATF-filtered tables so training and inference distributions match; the paper lists this as future work, and the observed in-domain drops would be the baseline to beat.","The fixed 40% row retention is the most task-blind component; a question-type-aware rule (keep all rows for aggregation or verification questions) is a testable modification that could recover the fact-verification losses.","Because the filtering scores come from an LLM, part of the out-of-domain gain may be an implicit ability of that LLM to recognize entities and schemas; swapping the scoring LLM for a weaker or stronger one would separate filtering skill from world knowledge.","The same describe-score-cluster-prune pattern could transfer to non-tabular structured inputs, such as JSON documents or knowledge-base excerpts, wherever a question targets a small subset of fields."],"forward_implications":["Models that already struggle with long tables can serve larger inputs: after ATF, the fraction of Open-WikiTable inputs exceeding TAPAS's length limit falls from 22.2% to 2.4%.","Out-of-domain TableQA improves without any parameter update: TAPAS EM rises from 0.599 to 0.610 on Open-WikiTable and from 0.376 to 0.508 on AIT-QA; TAPEX EM rises from 0.446 to 0.489 on Open-WikiTable.","ATF can narrow the model-size gap: TAPAS-base with ATF reaches 0.508 EM on AIT-QA, slightly above the 0.505 EM of unfiltered TAPAS-large.","Aggressive compression is safe for localized-reasoning QA but not for holistic tasks: table fact verification accuracy drops by 1.3-7.5 percentage points across TabFact and SEM-TAB-FACT even with 67-71% cell reduction.","The clustering-based column selection is load-bearing: replacing it with simple top-K selection drops EM and F1 by 5.7-8.0% on Open-WikiTable."],"supporting_citations":[{"why":"Defines TAPAS, the weak-supervision table-parsing baseline that ATF is evaluated on.","marker":"[Herzig et al., 2020]"},{"why":"Defines TAPEX, the second baseline QA model ATF is applied to.","marker":"[Liu et al., 2022]"},{"why":"Provides WikiTableQuestions, the in-domain TableQA dataset used for fine-tuning and evaluation.","marker":"[Pasupat and Liang, 2015]"},{"why":"Provides Open-WikiTable, the out-of-domain TableQA benchmark where ATF improves accuracy.","marker":"[Kweon et al., 2023a]"},{"why":"Provides AIT-QA, the hierarchical out-of-domain airline-industry benchmark where ATF produces large gains.","marker":"[Katsis et al., 2021]"},{"why":"Provides TabFact, the in-domain table fact verification dataset.","marker":"[Chen et al., 2020]"},{"why":"Provides SEM-TAB-FACT, the out-of-domain fact verification benchmark where ATF's accuracy drops.","marker":"[Wang et al., 2021]"},{"why":"Supplies the sentence embeddings used for dense column and row similarity scoring.","marker":"[Reimers and Gurevych, 2019]"}],"fun_headline_variants":["No-retrain table filter trims 70% of cells, improves unseen QA","Question-aware table pruning: 70% fewer cells, better cross-domain QA","ATF: Adaptive table filter learns what to keep, improves out-of-domain QA","Table QA prefilter cuts 70% of cells; gains on new formats, loses on fact-check","Keep only useful rows: table filter improves cross-domain QA, cuts cells 70%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every answer's evidence lies within the top 40% of rows (and the selected columns), so a question whose answer requires rows outside that fraction—such as an aggregation over most of the table—will have the needed evidence discarded before the QA model sees it.","fun_headline_variants_meta":{"raw":{"variants":["No-retrain table filter trims 70% of cells, improves unseen QA","Question-aware table pruning: 70% fewer cells, better cross-domain QA","ATF: Adaptive table filter learns what to keep, improves out-of-domain QA","Table QA prefilter cuts 70% of cells; gains on new formats, loses on fact-check","Keep only useful rows: table filter improves cross-domain QA, cuts cells 70%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001028,"raw_usage":{"total_tokens":4292,"prompt_tokens":867,"completion_tokens":3425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":3315}},"tokens_in":483,"tokens_out":3425,"duration_ms":23893,"temperature":1.0,"reasoning_tokens":3315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:41:13.890486+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ATF on an out-of-domain TableQA set where every question requires aggregating over more than 40% of the rows (for example, asking for a sum or average over all rows in the table); if accuracy drops below the unfiltered baseline, the claim that ATF preserves out-of-domain QA accuracy fails because the fixed row-retention rule systematically removes needed rows.","supporting_citations":[{"cited_title":"Tapas: Weakly supervised table parsing via pre-training","cited_arxiv_id":null,"evidence_quote":"Defines TAPAS, the weak-supervision table-parsing baseline that ATF is evaluated on."},{"cited_title":"Ait-qa: Question answering dataset over complex tables in the airline industry, 2021","cited_arxiv_id":null,"evidence_quote":"Provides AIT-QA, the hierarchical out-of-domain airline-industry benchmark where ATF produces large gains."},{"cited_title":"Tabfact: A large-scale dataset for table-based fact verification","cited_arxiv_id":null,"evidence_quote":"Provides TabFact, the in-domain table fact verification dataset."},{"cited_title":"Sentence-bert: Sentence embeddings using siamese bert-networks","cited_arxiv_id":null,"evidence_quote":"Supplies the sentence embeddings used for dense column and row similarity scoring."}],"review_version":1}