{"id":"57ee899b-480d-40d7-b61c-f102fd7fc2f6","arxiv_id":"2411.17110","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A class-aware LLM framework for example-driven table transformations that beats prior string-only and black-box methods on four benchmarks.","lead":"TabulaX is a framework that uses large language models to turn messy table columns into a clean, joinable format. It sorts transformations into four types, generates readable Python code or formulas for strings and numbers, and looks up external knowledge when needed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol does not establish generalization: test-set prompt selection, an in-context example overlapping TT, and unstated example-row exclusion make the reported F1 an upper bound.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation may not measure generalization because of test-set prompt selection, a potentially leaked few-shot example, and an unclear treatment of the n example rows. The paper's architecture diagram separates examples from source rows, so example exclusion may have been intended, but no sentence in Section 5.2 states that the examples are held out. The prompt-selection procedure is explicitly described, and the Gregorian-to-Jalali in-context example is explicitly described, but the overlap with the TT benchmark is not addressed. These are measurement issues rather than flaws in the underlying method; the framework is coherent and the numerical fitting, code generation, and classification components are plausible. The correct response is to keep the CONDITIONAL verdict and require a tightened, pre-registered evaluation protocol before the reported F1 numbers are treated as evidence of generalization to fresh table pairs.","tokens_in":20806,"tokens_out":3400,"duration_ms":34052,"concrete_test":"Re-run the full Table 3 evaluation under a locked protocol: (1) fix the classifier and per-class generator prompts on a validation split of table pairs (or on 10% of tables) before computing any test F1; (2) explicitly exclude the n example rows from the source rows scored for precision/recall and report both numbers; (3) remove the Gregorian-to-Jalali few-shot example from the algorithmic relationship tagger, or replace all in-context examples with pairs not appearing in any benchmark, and recompute TT and KBWT F1. If TT F1 drops below 0.85 or KBWT F1 drops below 0.45, the advertised advantage over DTT is substantially selection- or leakage-driven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an F1 comparison on four benchmarks, and three protocol choices make those numbers an upper bound on what a deployed system would achieve. First, Sections 4.2 and 5.2 state that 3–5 prompt variants were tested per class and the best one retained; because selection is done on the same benchmarks used for Table 3, there is no independent prompt-validation split, so the reported F1 can reflect test-set prompt selection. Second, Section 5.2 defines n input examples but never states whether those rows are removed from the source column before precision/recall are computed; if they are scored, a nontrivial fraction of rows (e.g., 5 of ~92 in WT) are trivially correct, directly inflating recall and F1. Third, Section 4.5 includes Gregorian-to-Jalali date conversion as an in-context training example for the relationship tagger; TT is an algorithmic-heavy benchmark and the paper itself mentions Gregorian-to-Hijri transformations, so this one example can leak the correct relationship for TT tables of that type. Any one of these could be benign if documented, but the paper is silent on the first two and the third is acknowledged as a training example without noting the benchmark overlap. Since the headline gaps over DTT and GXJoin are large on TT and KBWT, the protocol must be tightened before the generalization claim is accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TabulaX, a framework that uses large language models for multi-class column-level table transformations. A classifier routes input column pairs into four classes—string-based, numerical, algorithmic, and general—and each class is handled by a tailored module: Python function generation for string transformations, curve fitting with Levenberg-Marquardt for numerical transformations, chain-of-thought relationship tagging plus code generation for algorithmic transformations, and LLM-constructed lookup for general transformations. The system is evaluated on four benchmarks (WT, SS, TT, KBWT) against DTT, GXJoin, AFJ, and Explain-Da-V, reporting F1 scores of 0.983, 0.952, 0.918, and 0.567 respectively, with interpretable transformation functions for all classes except general. The authors also release the code and datasets.","tokens_in":21097,"tokens_out":4161,"duration_ms":39883,"significance":"If the reported results are trustworthy, the paper makes a useful empirical contribution: it shows that an off-the-shelf LLM, combined with lightweight class routing and module-specific prompting, can outperform specialized programming-by-example and transformer baselines on heterogeneous join tasks while emitting human-readable Python transformations. The explicit release of code/datasets, the runtime analysis comparing LLM call counts, and the candid discussion of the general-transformation class limitations are strengths. However, the central claim is an F1 comparison, and the evaluation protocol as currently described does not establish that the reported numbers measure generalization on fresh table pairs.","major_comments":[{"comment":"Prompt selection is performed on the same benchmarks used for the reported F1 scores. Section 5.2 states that \"For each transformation class, we tested 3–5 prompt variants and selected the best-performing one for our experiments,\" and Section 4.2 similarly says the authors \"try 3-5 prompts before selecting one.\" No independent prompt-validation split is described. Under this protocol, the reported F1 is an optimistic upper bound that includes test-set prompt selection. Please report results for all prompt variants, or use a held-out prompt-development set and report only the F1 obtained by the prompt selected without access to the test benchmarks.","section":"§5.2, Table 3"},{"comment":"The paper defines n input examples but never states whether those n rows are removed from the source column before precision, recall, and F1 are computed. If they are not removed, the TT dataset (average about 8 rows, n=3) has up to 37.5% of its rows trivially correct because the example rows are given in the prompt, and WT/KBWT (n=5 of roughly 92–113 rows) receive a recall boost of about 4–6%. This bears directly on the headline TT gap (0.918 vs. 0.643 for DTT). Please clarify the scoring protocol explicitly and report results both with and without the example rows included.","section":"§5.2, Table 3"},{"comment":"The relationship-tagging prompt for algorithmic transformations includes an in-context example of Gregorian-to-Jalali date conversion. The TT benchmark is predominantly algorithmic, and Section 5.6 itself discusses the difficulty of Gregorian-to-Hijri date transformations in the algorithmic class. If TT contains date-conversion tables of this type, the supplied few-shot example directly reveals the expected transformation relationship, inflating the algorithmic-class results on TT. Please either remove this overlapping example from the deployed prompt and rerun the TT experiments, or provide a per-relationship breakdown showing that TT date-conversion tables are not benefiting from the in-context example.","section":"§4.5, §5.6"}],"minor_comments":[{"comment":"The support counts in the class-wise tables sum to 448 (207+60+105+76), while the dataset descriptions in Section 5.1 sum to 450 table pairs (31+108+230+81). Please reconcile these counts or explain the discrepancy.","section":"Tables 2 and 5"},{"comment":"The GPT-4o rows in Table 6 are identical to the \"Golden\" classification rows in Table 4, yet Section 5.7 presents Table 6 as an evaluation of different model sizes without stating which classification setting is used. Please state explicitly whether the model-size experiments use the LLM classifier or the golden classifier.","section":"Table 6"},{"comment":"Equation (1) is not syntactically well-formed: the expression \"∀s_j∈S'((s_j, f(s_j)∈ E)\" mixes set membership and the tuple notation, and the intended condition is unclear. Please rewrite it as a clean formal statement.","section":"Eq. (1)"},{"comment":"There is a grammatical error in the sentence \"the generated code may is incomplete,\" and the contributions section contains the typo \"reproducability.\" These should be corrected.","section":"§5.6"},{"comment":"The comparison \"TDE reports a coverage of 72% on TT, whereas our method achieves a recall of 92%\" compares two different metrics; coverage and recall are not interchangeable. Please clarify how the TDE coverage number translates to the precision/recall framework used for the other baselines.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The three major concerns are all fixable within the scope of the manuscript: report a prompt-validation split, clarify and re-run with example-row exclusion, and address the possible in-context overlap with TT. If the authors can show that the reported numbers are robust to these protocol changes, the paper would be a solid contribution. I do not see grounds for rejection if the central empirical claims survive the tightened protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TabulaX is a real step beyond the authors' own DTT and GXJoin: instead of one string-transformation engine, it routes each table pair through a classifier to a specialized module — numeric curve fitting, LLM code generation for strings, chain-of-thought relationship tagging for algorithmic cases, and LLM-built lookup for general cases. That combination is new, the interpretable Python output is a genuine advantage over DTT's latent space, and the paper is honest about the general class's limitations (hallucination risk, row-by-row LLM calls, non-interpretability). They also release code, evaluate three LLM backends, and compare against four baselines. Credit where due: the architecture is sensible and the limitation paragraph is candid.\n\nNow the soft spots, in proportion. The evaluation protocol as written does not establish generalization, and the problem is concentrated in three unspecified choices. First, Section 5.2 never states whether the n provided example rows are removed from the source column before precision/recall are computed. This matters a lot: on TT, n=3 out of an average of 8 rows means nearly 40% of the evaluation could be trivially correct. For WT it's 5 of ~92 rows — smaller, but still an inflation. Second, Sections 4.2 and 5.2 report testing 3–5 prompt variants per class and keeping the best, on the same benchmarks used for the headline table. The paper says performance was stable across prompts, which mitigates the concern, but it doesn't quantify that stability or use a validation split. Third, the in-context example for relationship tagging includes Gregorian-to-Jalali conversion; TT contains Gregorian-to-Hijri tasks, so that training shot overlaps the test distribution. Any one of these might be benign if documented; all three together mean the reported F1 gaps over DTT on TT and KBWT should be read as upper bounds.\n\nTwo minor notes: there are no error bars or repeated-run variance, and the numerical module fits and evaluates on the same tables. The \"Golden classifier\" comparison in Table 4 is a nice control but doesn't resolve the hold-out question.\n\nThe central design contribution survives these concerns — the framework is plausible and the qualitative advantages are real. But the quantitative claims need a protocol fix before they're trustworthy.\n\nWho it's for: the data integration and programming-by-example community, and anyone building LLM-based table transformation tools. It deserves serious peer review, and I'd push for a revision that clarifies example-row exclusion, reports scores with examples removed, and either adds a prompt-validation split or reports all prompt variants.","headline":"A genuinely useful multi-class table transformation framework with an evaluation protocol that, as written, makes the headline F1 numbers upper bounds rather than reliable estimates.","tokens_in":21643,"tokens_out":2366,"would_cite":true,"duration_ms":25689,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a framework which classifies table mismatches into four types and then applies a tailored LLM generator beats specialized string-only and fuzzy-join systems on four benchmarks while returning human-readable Python or…","keywords":["tabular data transformation","heterogeneous table join","large language models","programming by example","interpretable transformations","chain-of-thought prompting","data integration","multi-class classification"],"falsifier":"Hold out the n example rows from the F1 computation and lock the prompt for each class before seeing the four benchmarks; if the TT and KBWT F1 gaps over DTT and GXJoin collapse, the claim that TabulaX's accuracy reflects generalization rather than selection and benchmark-derived few-shot examples is false.","tokens_in":20556,"feed_emoji":"🔀","tokens_out":8714,"duration_ms":72949,"temperature":0.7,"pith_summary":"TabulaX sets out to show that example-driven table transformation, converting values in a source column into the format of a target column so two tables can be joined, can be handled across a wide range of mismatch types by first classifying the mismatch and then applying a tailored generator. The four classes are string-based, numerical, algorithmic, and general (requiring outside knowledge), and for the first three the framework emits an explicit Python function or numeric formula that a user can inspect, verify, and edit. Only the general class relies on opaque LLM predictions over a lookup constructed from the model's knowledge. On four public benchmarks the framework reports F1 scores of 0.983, 0.952, 0.918, and 0.567, ahead of the string-only and fuzzy-join baselines it compares against. If correct, this means a general-purpose LLM plus a small classification layer can cover a broader class of transformations than specialized search engines while returning interpretable mappings.","feed_headline":"LLM framework joins mismatched tables at up to 98.3% F1","feed_subtitle":"TabulaX classifies each column pair as string, numeric, algorithmic, or general, then emits Python or a formula users can inspect.","key_machinery":"The load-bearing mechanism is the class-routing architecture: a serialization convention that presents example pairs as source-arrow-target strings, an LLM classifier that dispatches each table pair to one of four modules, and modules specialized per class. Numeric transformations are handled by Levenberg-Marquardt curve fitting over four function families, with the lowest-MSE function selected and then rendered as Python. String-based transformations use an LLM prompted with Input and Expected output test cases to write a Python function, verified by a syntax checker. Algorithmic transformations split the task into relationship tagging, aided by chain-of-thought prompting, followed by Python generation. General transformations use column-type detection followed by an LLM-constructed lookup, with the known weaknesses of hallucination, per-row cost, and non-interpretability. Downstream joining tolerates small mismatches by edit-distance matching for text and absolute difference for numbers.","core_discovery":"The paper's central discovery claim is that the difficulty in table transformation lies less in finding a transformation than in knowing which kind of transformation is needed, and that a classifier routing each table pair to a dedicated generator solves the multi-class problem better than a single model trying to do everything. Given a small set of matched rows, the classifier assigns the pair to one of four classes; numeric pairs go to curve fitting with mean-square-error selection among linear, polynomial, exponential, and rational forms; string pairs go to LLM-written Python code; algorithmic pairs go through a chain-of-thought relationship tagger and then Python generation; general pairs go to an LLM-based lookup. The framework reports that this design consistently achieves higher or comparable F1-scores than DTT, GXJoin, AFJ, and Explain-Da-V on the WT, SS, TT, and KBWT benchmarks, and that it produces interpretable transformation functions for all classes except general. It also reports that the classifier, although imperfect (macro F1 of 0.84 with its best model), costs little end-to-end: results with ground-truth class labels are close to those with LLM classification.","pith_inferences":["My inference: the reported gains should be retested with prompts frozen per class before seeing the benchmarks and with the n example rows excluded from scoring; until then the headline F1 gap over DTT and GXJoin may partly reflect tuning rather than transfer.","My inference: the class-routing design extends naturally to multi-column transformations, where each column pair is classified and transformed separately and the results are reconciled, a combination the paper leaves for future work.","My inference: the general class could be made both more accurate and more scalable by replacing the LLM lookup with retrieval from a knowledge base or bridge table, directly addressing the hallucination and per-row-cost limitations the paper lists."],"forward_implications":["String-only baselines can be outperformed on mixed benchmarks by adding a numeric curve-fitting module and a chain-of-thought algorithmic generator rather than relying on one model to handle every case.","Interpretable transformation functions are achievable without a domain-specific language: Python functions and numeric formulas can be read, edited, and executed once per column for string, numeric, and algorithmic classes.","The general class is the remaining accuracy bottleneck, with F1 of 0.567 on KBWT, driven by the model's internal knowledge limits and by one-to-many mappings.","Edit-distance matching buys a large recall increase with minimal precision loss, so the same generated functions support both heterogeneous joins and exact-match tasks such as imputation."],"supporting_citations":[{"why":"Supplies the main LLM baseline (DTT) and the string-only limitation TabulaX is designed to surpass.","marker":"[8]"},{"why":"Supplies GXJoin, the best interpretable string-based baseline, and its exhaustive-search transformation space.","marker":"[34]"},{"why":"Supplies AFJ, the fuzzy-similarity baseline that TabulaX compares against on all four datasets.","marker":"[25]"},{"why":"Supplies Explain-Da-V, the structural table-reshaping baseline used for comparison.","marker":"[39]"},{"why":"Introduces the Web Tables (WT) benchmark and the Auto-join string-transformation approach the dataset is built to stress.","marker":"[53]"},{"why":"Contributes FlashFill and the spreadsheet examples that form part of the SS benchmark.","marker":"[14]"},{"why":"Contributes BlinkFill and its spreadsheet transformations to the SS benchmark.","marker":"[40]"},{"why":"Introduces the TT dataset and the TDE search engine whose 72 percent coverage is cited as a comparison point.","marker":"[18]"},{"why":"Introduces DataXFormer and the KBWT dataset of knowledge-base-derived tables needing external knowledge.","marker":"[1]"},{"why":"Provides the chain-of-thought prompting technique used by the algorithmic and general relationship taggers.","marker":"[47]"}],"fun_headline_variants":["LLM routes column pairs to four transform types, hits 98.3% F1","TabulaX: LLM picks transform type, then writes interpretable code","LLM classifies table pairs into four types, then emits Python or formula","Four-way table transform: LLM classifier enables 98.3% F1","TabulaX: LLM-driven multi-class table transformations with code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that choosing the best of 3-5 prompt variants per class, using a few-shot example drawn from the benchmark being tested, and not excluding the n example rows from the reported F1 still yields a fair measure of how TabulaX generalizes to unseen table pairs.","fun_headline_variants_meta":{"raw":{"variants":["LLM routes column pairs to four transform types, hits 98.3% F1","TabulaX: LLM picks transform type, then writes interpretable code","LLM classifies table pairs into four types, then emits Python or formula","Four-way table transform: LLM classifier enables 98.3% F1","TabulaX: LLM-driven multi-class table transformations with code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001196,"raw_usage":{"total_tokens":4935,"prompt_tokens":954,"completion_tokens":3981,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":3878}},"tokens_in":570,"tokens_out":3981,"duration_ms":26911,"temperature":1.0,"reasoning_tokens":3878,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:30:25.401799+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out the n example rows from the F1 computation and lock the prompt for each class before seeing the four benchmarks; if the TT and KBWT F1 gaps over DTT and GXJoin collapse, the claim that TabulaX's accuracy reflects generalization rather than selection and benchmark-derived few-shot examples is false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the main LLM baseline (DTT) and the string-only limitation TabulaX is designed to surpass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies GXJoin, the best interpretable string-based baseline, and its exhaustive-search transformation space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies AFJ, the fuzzy-similarity baseline that TabulaX compares against on all four datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Explain-Da-V, the structural table-reshaping baseline used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Web Tables (WT) benchmark and the Auto-join string-transformation approach the dataset is built to stress."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes FlashFill and the spreadsheet examples that form part of the SS benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes BlinkFill and its spreadsheet transformations to the SS benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the TT dataset and the TDE search engine whose 72 percent coverage is cited as a comparison point."},{"cited_title":"Ilyas, Mourad Ouzzani, Paolo Papotti, and Michael Stonebraker","cited_arxiv_id":null,"evidence_quote":"Introduces DataXFormer and the KBWT dataset of knowledge-base-derived tables needing external knowledge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the chain-of-thought prompting technique used by the algorithmic and general relationship taggers."}],"review_version":1}