{"id":"fa894171-287c-4ede-9ca0-daee4b7e642e","arxiv_id":"2507.03971","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Continuing the pre-training of TabPFN on 71 curated real-world tables raises its average normalized ROC-AUC from 0.954 to 0.976 on 29 AutoML Benchmark datasets.","lead":"Researchers continued training the tabular AI model TabPFN on 71 curated real-world datasets from OpenML and Kaggle, improving its average accuracy on 29 standard benchmark tasks. The improved model, Real-TabPFN, is meant to serve as a stronger off-the-shelf baseline for small table classification problems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contamination filter's leakage-freedom claim is asserted, not demonstrated; the 71 OpenML pre-training datasets and 29 AMLB eval datasets share a common source, and the paper reports no exclusion counts, dataset IDs, or hash-verification evidence.","rationale":"I read the paper in good faith. It is a short empirical workshop paper with a clear central claim: continued pre-training of TabPFNv2 on 71 curated real-world datasets yields a better default tabular classifier on the 29 AMLB datasets. The experiment is well-controlled in one sense (same base checkpoint, same eval protocol, and a statistically significant Wilcoxon test), and the reported gain is modest and plausible. However, the single most load-bearing premise is that the evaluation datasets are absent from the pre-training corpus. The paper's Section 3 contamination section is descriptive rather than evidential: it lists five filtering tiers but provides no exclusion counts, no dataset IDs, no hash values, and no per-dataset verification. The overlap risk is real because both pre-training and evaluation data come from OpenML, and OpenML is known to contain many versions, renamed copies, and subsample/superset relations of the same underlying datasets. The filter's first criterion (pre-training datasets >10,000 samples) does not prevent leakage because evaluation datasets can appear as subsets of larger pre-training datasets or as supersampled versions, and the paper itself acknowledges that AMLB datasets are derived from larger sources (Gijsbers et al., 2023). The claimed improvement is small in raw terms (0.929 to 0.932 ROC-AUC), so even a modest overlap could plausibly account for the entire observed gain or the statistical significance. I do not allege dishonesty; the authors may well have done the manual inspection carefully. The problem is that the manuscript does not provide the evidence needed to verify it, which is exactly what the reader's CONDITIONAL verdict requires. I also note the conclusion's overreach when it claims Real-TabPFN outperforms every baseline on all evaluated datasets—Appendix C shows it does not dominate all baselines on all datasets (e.g., tuned AutoGluon is close, and several default baselines are competitive on specific datasets). But the leakage concern is the more fundamental one, so the verdict remains CONDITIONAL with the condition that the corpus and contamination checks be published independently.","tokens_in":10790,"tokens_out":1939,"duration_ms":17646,"concrete_test":"Release the full list of the 71 pre-training dataset IDs and the 29 evaluation dataset IDs, then run an independent automated contamination check: for each eval dataset, compute row/column hashes and compare against every pre-training dataset, including all OpenML versions/parquet snapshots and their train/test splits; also check for approximate subset/superset relations and renamed variants. Report per-dataset exclusion counts and hash-match results. If any eval dataset matches a pre-training dataset, re-run the headline comparison excluding that dataset to see if the 0.954-to-0.976 gain and Wilcoxon p=0.0045 survive.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim is that Real-TabPFN's improved mean normalized ROC-AUC (0.954 to 0.976, Wilcoxon p=0.0045) over TabPFNv2 is due to continued pre-training on curated real-world data. This claim collapses if any of the 29 evaluation datasets, or a superset, subsample, or renamed variant, appears in the 71-dataset pre-training corpus. Section 3 describes a five-tier contamination filter but gives no quantitative outcome: no exclusion counts, no dataset IDs, no hashes, no per-dataset check results. Both corpora come from OpenML, and the filter's criterion (1) (only datasets >10,000 samples) is not itself sufficient to prevent leakage because AMLB datasets have subsampled/supersampled versions on OpenML (e.g., many AMLB tasks are derived from larger OpenML source datasets, and OpenML hosts duplicated and modified versions of the same underlying data under different IDs and names). The paper asserts manual inspection and row/column hash checks, but the absence of leakage is an assumption, not a demonstrated fact; the reported gain is the difference between the model that saw the eval data and one that did not, so a single leaked dataset could drive the measured improvement. This is the load-bearing weak point: everything else in the paper depends on the comparison being contamination-free, and the paper provides no verifiable evidence for it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes Real-TabPFN, obtained by continued pre-training of TabPFNv2 on 71 curated real-world datasets from OpenML and Kaggle, using a low learning rate, AdamW with cosine scheduling, and L2-SP regularization. Evaluation on 29 OpenML AutoML Benchmark classification tasks using 10-fold cross-validation reports mean normalized ROC-AUC improving from 0.954 (TabPFNv2 default) to 0.976, with a two-sided Wilcoxon signed-rank p = 0.0045. Additional ablations examine context size and data-source choices, claiming larger contexts and the union of OpenML and Kaggle corpora give the best results.","tokens_in":10953,"tokens_out":5584,"duration_ms":58217,"significance":"The paper's central empirical result, if contamination-free, is practically valuable: it demonstrates that a small, curated real-world corpus can improve a state-of-the-art synthetic-only tabular foundation model, yielding a stronger off-the-shelf baseline. The headline comparison is well controlled in design (same architecture, same base checkpoint, same evaluation protocol, and the authors commit to releasing the model weights), and the use of a Wilcoxon test on paired data is appropriate in principle. However, the main claim rests on an asserted but undocumented absence of train/evaluation overlap, and the secondary ablations lack any uncertainty quantification. The contribution is promising but not yet verified to the standard required for the claims made.","major_comments":[{"comment":"The five-tier contamination filter is described but its outcomes are never reported: there are no exclusion counts, no OpenML IDs for the 71 training datasets, and no per-dataset hash or name-matching results. Because both the pre-training and evaluation corpora are drawn from OpenML, and AMLB datasets are known to have supersets, subsamples, and duplicated uploads on OpenML, criterion (1) (only datasets with more than 10,000 samples) does not by itself rule out overlap. The statement 'We exclude any dataset from the pre-training data that does not meet these criteria' is an assertion, not evidence. This is load-bearing: a single leaked evaluation dataset, or a superset of one, could drive the reported difference between 0.954 and 0.976. Please provide the full training dataset list with OpenML IDs, the exclusion log (counts per filter stage), the hash-verification records against the 29 evaluation tasks, and a leakage sensitivity analysis (e.g., training only on the Kaggle subset and showing a similar result).","section":"Section 3, Appendices A-B"},{"comment":"All ablation results are single-seed point estimates with no error bars, repeated runs, or significance tests. For example, Figure 4 reports +0.010 for both 2048 and 4096 context sizes, and Figure 5 reports +0.015 (Kaggle), +0.019 (OpenML), and +0.022 (Both); these differences are small relative to the noise expected from a single continued pre-training run of 20,000 steps with batch size 1. Without multiple seeds or variance estimates, the ranking of context sizes and the conclusion that the union of sources is 'best' are not established. This does not invalidate the headline comparison, but it is load-bearing for the paper's secondary conclusions about why the method works.","section":"Section 5, Figures 4-6"},{"comment":"The Wilcoxon signed-rank test reported in Figure 1 is a paired test, but the manuscript does not state whether Real-TabPFN and TabPFNv2 were evaluated on the exact same 10-fold splits. If the fold splits differ between the two models, the pairing assumption is violated and the p-value of 0.0045 could overstate significance. Please state explicitly whether the fold indices are identical for both models; if they are not, rerun the comparison on shared folds or report an unpaired test such as the Mann-Whitney U test.","section":"Section 5, Figure 1"}],"minor_comments":[{"comment":"The text and figures use inconsistent corpus names: 'GitTable' versus 'GitTables' and 'CommonCrawl' versus 'Common Crawl'.","section":"Figure 2 and Section 3"},{"comment":"The caption contains a typo: 'Comparsion' should be 'Comparison'.","section":"Figure 3 caption"},{"comment":"The L2-SP penalty is defined as the squared L2 norm of (w - w0); please call it the 'squared L2 norm' rather than 'L2 norm' to match the equation.","section":"Section 4"},{"comment":"Provide OpenML dataset IDs for all OpenML entries; the current name-only list includes ambiguous entries (e.g., 'Bank marketing data set UCI' and 'Bank Marketing') that make it impossible for readers to verify the contamination filter.","section":"Appendix A"},{"comment":"The sentence 'outperforms every other state-of-the-art baseline on all evaluated datasets' overstates the evidence in Table C, which shows the highest mean normalized ROC but not per-dataset dominance over every baseline; please rephrase to 'on average'.","section":"Section 6"},{"comment":"The reference for Vanschoren et al. contains a truncated URL/DOI ('264119.'); please correct it.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from a clear data-availability statement and a full audit trail for the contamination filter. If the authors can provide the OpenML IDs, exclusion counts, and hash-verification evidence, the main claim becomes verifiable and the paper could be acceptable. I would not accept the manuscript without these items."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short empirical workshop paper: continue TabPFNv2's pre-training on 71 curated OpenML/Kaggle tables with a low learning rate and L2-SP regularization, then evaluate on the same 29 AMLB tasks. The headline finding is believable: Real-TabPFN improves mean normalized ROC from 0.954 to 0.976, Wilcoxon p=0.0045, and the comparison is well controlled because the base checkpoint, architecture, and evaluation protocol are identical to TabPFNv2. The two-stage synthetic-then-real recipe is genuinely new relative to the cited work, and the ablations showing that larger contexts and the union of OpenML+Kaggle help are informative.\n\nThe soft spot is the one the stress-test calls out, and it is load-bearing. The entire gain is the difference between a model that saw the evaluation-distribution data and one that did not. Both the 71 training tables and the 29 evaluation tasks come from OpenML, and the paper's five-tier contamination filter is described only qualitatively: no exclusion counts, no dataset IDs, no hash-verification records. The claim that no evaluation dataset, or a subsample or renamed variant of one, appears in training is asserted, not demonstrated. That has to be fixed with an actual audit before the result is trustworthy.\n\nThe rest is minor. The ablations are single-run point estimates without error bars. The raw ROC gain is small (0.929 to 0.932), so the practical impact is modest. The conclusion's sentence about outperforming every state-of-the-art baseline on all evaluated datasets overstates what a mean normalized score shows. And the paper does not list dataset IDs or provide code in the text, though the weights are promised.\n\nAudience: people working on tabular foundation models, especially those building on TabPFN. It is a workshop-level contribution, not a Nature-scale claim, but it is a legitimate empirical step. My recommendation: send it to review, and make acceptance conditional on publishing the pre-training dataset IDs, the contamination audit, and variance estimates for the ablations. The central idea is worth taking seriously.","headline":"A credible, well-controlled continued pre-training result whose main claim hinges on a contamination audit the paper does not actually provide.","tokens_in":11667,"tokens_out":1609,"would_cite":true,"duration_ms":15302,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Continued pre-training of the TabPFNv2 checkpoint on 71 curated real-world tables raises mean normalized ROC-AUC from 0.954 to 0.976 across the 29 small classification datasets of the OpenML AutoML Benchmark.","keywords":["tabular foundation models","continued pre-training","in-context learning","TabPFN","real-world tabular data","L2-SP regularization","OpenML AutoML Benchmark","data contamination"],"falsifier":"Publish the OpenML IDs and row/column hashes of the 71 training tables and check each of the 29 evaluation datasets, together with any superset, subsample, or renamed variant of it, against that list: if any evaluation task matches a training table, the 0.022 mean normalized ROC-AUC gain is contaminated and the claim falls. A second control is to retrain under the same recipe after deleting every training table from the same dataset family as any evaluation task and compare the resulting normalized ROC-AUC.","tokens_in":10448,"feed_emoji":"📊","tokens_out":11013,"duration_ms":91687,"temperature":0.7,"pith_summary":"This paper claims that a tabular foundation model pre-trained purely on synthetic data can be improved by a second, targeted pre-training phase on real-world tables. Starting from the TabPFNv2 checkpoint, the authors continue training for 20,000 steps on 71 curated OpenML and Kaggle datasets with a low learning rate and an L2-SP regularizer that keeps the weights near their starting point. On the 29 OpenML AutoML Benchmark classification tasks used in TabPFNv2's own evaluation, the resulting model, Real-TabPFN, raises mean normalized ROC-AUC from 0.954 to 0.976, a difference reported as significant at $p = 0.0045$ under a two-sided Wilcoxon signed-rank test. The authors also show that larger contexts during continued pre-training give larger gains and that a small curated corpus beats much larger web corpora such as CommonCrawl and GitTables. If correct, the result gives practitioners a stronger default model for small tabular classification at no extra inference cost, and it identifies curated real data, rather than more synthetic data, as the cheapest route to improving such models.","feed_headline":"0.976: real-world tables beat synthetic-only pre-training for TabPFN","feed_subtitle":"A 20,000-step continued pre-training on 71 curated OpenML and Kaggle tables lifts accuracy on small benchmarks.","key_machinery":"The load-bearing mechanism is two-stage pre-training guarded by an L2-Starting-Point (L2-SP) regularizer. Stage 1 provides the TabPFNv2 transformer trained on more than 100 million synthetic tables; Stage 2 continues training on real tables while the penalty $\\Omega(w) = \\frac{\\alpha}{2}\\lVert w - w_0\\rVert_2^2$, with $\\alpha = 0.003$, penalizes squared distance from the initial weights $w_0$ and thereby curbs catastrophic forgetting. The low learning rate, linear warm-up with cosine annealing, batch size of one dataset, and the 60/40 context/query split let each real table enter as an in-context learning episode, so the improvement reflects a shift in the model's prior toward real-world table statistics rather than task-specific fine-tuning.","core_discovery":"The central discovery is that the synthetic-to-real gap in tabular foundation models can be closed by a short continued pre-training phase on a modest number of large, curated real-world tables. Keeping the TabPFNv2 architecture fixed, the authors continue pre-training on 71 datasets from OpenML and Kaggle, capped per table at 20,000 rows and 400,000 cells, with batch size 1, a 60% context / 40% query split, a learning rate of $3\\times 10^{-7}$, and an L2-SP penalty with strength $\\alpha = 0.003$ that pulls the weights back toward the synthetic checkpoint. Evaluated on the 29 AutoML Benchmark classification tasks below 10,000 samples and 500 features, Real-TabPFN improves mean normalized ROC-AUC from 0.954 to 0.976, and the paper reports a two-sided Wilcoxon signed-rank $p$ of 0.0045 against default TabPFNv2. The paper further reports that the gain grows with continued-pretraining context size, that OpenML and Kaggle sources are complementary (+0.019 and +0.015 alone, +0.022 combined), and that CommonCrawl and GitTables corpora yield smaller or negative gains, supporting curation over breadth.","pith_inferences":["The reported gain is an upper bound until leakage is ruled out: the paper asserts a five-tier contamination filter without reporting exclusion counts or per-dataset verification, and both the 71 training tables and the 29 evaluation tasks come from OpenML, so a released list of training dataset IDs and hashes would settle whether any evaluation task, superset, subsample, or renamed variant appears","The recipe is portable to future synthetic-pretrained tabular foundation models: the same low-rate, L2-SP-protected second phase could be applied to models that handle larger tables, and the paper's context-size curve predicts the gain would grow there.","The negative CommonCrawl result suggests a floor effect: small, noisy tables can actively degrade the learned prior, implying that table-size filtering should precede quality filtering when building tabular pre-training corpora, and that per-source ablation curves are a cheap screening test for candidate corpora.","A natural testable extension is regression and survival targets: the paper evaluates classification only, and it is an open question whether the 0.022 gain carries over when the loss and metric change."],"forward_implications":["Real-TabPFN becomes a stronger off-the-shelf baseline for small tabular classification than default TabPFNv2, with essentially the same inference cost (about 2.9 seconds per dataset in the paper's runs).","Continued-pretraining gains grow with context size, so the recipe has headroom: any increase in the 20,000-row or 400,000-cell cap should translate into further accuracy gains.","A small curated corpus of 71 large tables outperforms roughly 97,000 filtered CommonCrawl and 658 GitTables datasets, so corpus quality and table size dominate corpus breadth for this stage.","OpenML and Kaggle data are complementary: combining them gives +0.022 normalized ROC-AUC versus +0.015 or +0.019 alone, so pooling heterogeneous real sources is the better recipe.","Because Real-TabPFN beats every tuned baseline (AutoGluon, XGBoost, CatBoost, LightGBM, Random Forest, SVM, MLP, logistic regression) on mean normalized ROC-AUC, the default model becomes the recommended first call on small tabular problems."],"supporting_citations":[{"why":"Supplies the TabPFNv2 checkpoint and the exact 29-dataset evaluation protocol that Real-TabPFN is measured against.","marker":"Hollmann et al. (2025)"},{"why":"Establishes the synthetic-data pre-training paradigm that Stage 1 of the recipe builds on.","marker":"Hollmann et al. (2023)"},{"why":"Introduces the L2-SP regularizer that the authors add to the pre-training objective to protect the base weights.","marker":"Li et al. (2018)"},{"why":"Motivates the regularizer by documenting catastrophic forgetting in neural networks.","marker":"Kirkpatrick et al. (2017)"},{"why":"Defines the OpenML AutoML Benchmark from which the 29 evaluation datasets are taken.","marker":"Gijsbers et al. (2023)"},{"why":"Provides the OpenML repository that contributes a large share of the 71 continued-pre-training datasets.","marker":"Vanschoren et al. (2013)"},{"why":"Demonstrates continued pre-training as an effective paradigm in language models, the idea this paper transfers to tabular data.","marker":"Gururangan et al. (2020)"},{"why":"Supplies the cosine annealing schedule used with the low learning rate in Stage 2.","marker":"Loshchilov & Hutter (2017b)"}],"fun_headline_variants":["Real-world pre-training lifts TabPFN to 0.976 AUC","Curated real tables boost tabular foundation model accuracy","Short real-data pre-training boosts TabPFN on small benchmarks","TabPFN improves via continued pre-training on curated real data","Real-TabPFN: 20k steps on 71 curated tables gain 0.022 AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the assumption that the five-tier contamination filter fully removed every overlap between the 71 pre-training tables and the 29 evaluation tasks, even though both corpora come from OpenML and the paper reports no exclusion counts or per-dataset verification.","fun_headline_variants_meta":{"raw":{"variants":["Real-world pre-training lifts TabPFN to 0.976 AUC","Curated real tables boost tabular foundation model accuracy","Short real-data pre-training boosts TabPFN on small benchmarks","TabPFN improves via continued pre-training on curated real data","Real-TabPFN: 20k steps on 71 curated tables gain 0.022 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000712,"raw_usage":{"total_tokens":3184,"prompt_tokens":909,"completion_tokens":2275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2180}},"tokens_in":525,"tokens_out":2275,"duration_ms":15726,"temperature":1.0,"reasoning_tokens":2180,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:58:41.836479+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Publish the OpenML IDs and row/column hashes of the 71 training tables and check each of the 29 evaluation datasets, together with any superset, subsample, or renamed variant of it, against that list: if any evaluation task matches a training table, the 0.022 mean normalized ROC-AUC gain is contaminated and the claim falls. A second control is to retrain under the same recipe after deleting every training table from the same dataset family as any evaluation task and compare the resulting normalized ROC-AUC.","supporting_citations":[],"review_version":1}