{"id":"8ce4a486-7b15-4932-8f42-f5ca29686a85","arxiv_id":"2607.26000","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"All nine tested tabular foundation models degrade under distribution shift, and real-world pretraining provides no robustness advantage over synthetic pretraining.","lead":"This paper tests nine tabular foundation models—TabPFN, TabICL, Mitra, and others—on out-of-distribution data from three real-world datasets. All models lost accuracy under distribution shift, and pretraining on real data did not make them more robust.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported negative shift gaps in Fig. 3 contradict the claim that all TFMs degrade systematically; central claim needs qualification.","rationale":"The concern is load-bearing because the paper's headline and conclusions depend on universal degradation; a single counterexample in the same paper's data defeats the universal. The reader's weakest_assumption focused on representativeness and default configurations, which are external validity concerns; the more direct issue is internal consistency of the reported aggregate. Because this is fixable by qualification and does not necessarily invalidate the broader finding that most TFMs degrade and that ID/OOD relationship is roughly proportional, I would not escalate to REJECT; keeping the reader's CONDITIONAL verdict (communicated as UNCHANGED) with the requirement to reconcile negative gaps is appropriate.","tokens_in":7779,"tokens_out":4825,"duration_ms":44065,"concrete_test":"From saved per-seed predictions (or rerunning the three TableShift experiments), compute ΔROC-AUC = ID AUC − OOD AUC for every model–dataset pair. (1) Count pairs with negative mean Δ and seeds with negative Δ; (2) overlay all points on the Fig. 2 scatter to confirm whether any lie above the diagonal; (3) if any negative gap exists, revise the Abstract/§3.2/§5 to replace 'all TFMs degrade systematically' with 'most TFMs degrade, with occasional small OOD improvements (especially synthetic/mixed priors)', and update the shift-gap range to include the negative values. This single check settles whether the universal claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim — 'all evaluated TFMs degrade systematically under distribution shift regardless of pre-training strategy' (Abstract; repeated in §3.2 and §5) — is contradicted by the paper's own Fig. 3 description. In §3.2, the text states: 'Both Mixed and Synthetic include negative gap values, meaning some models in these groups marginally improve under OOD conditions on certain dataset.' A negative ΔROC-AUC means OOD ROC-AUC exceeds ID ROC-AUC, so at least one model–dataset pair does not degrade; the Fig. 2 claim that 'no model achieving OOD parity with its ID performance across any dataset' cannot be true. This is not an external representativeness issue: it is an internal inconsistency in the reported evidence. The universal quantifier in the central claim is falsified by the authors' own aggregate. A related inconsistency appears when the same section reports largest degradation on Childhood Lead but concludes 'label shift is the most relevant shift type', although HELOC is the label-shift dataset and is described as moderate. Because the headline assertion is an unconditional universal, one negative gap is sufficient to require a revised, qualified conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical benchmark of nine tabular foundation models (TabPFN variants, TabICL variants, Mitra, LimiX, TabFM) on three TableShift datasets covering label, socioeconomic, and geographic distribution shifts. The authors report ID and OOD ROC-AUC for each model-dataset pair, define a shift gap, and claim that all evaluated TFMs degrade systematically under distribution shift, that real-world pre-training does not confer OOD robustness, that ID-OOD performance relationships known for classical models extend to TFMs, and that scalability limits several high-performing models. The results are based on five random seeds and default inference configurations.","tokens_in":7985,"tokens_out":4034,"duration_ms":41683,"significance":"If the central claim were fully supported, this would be a useful early benchmark for OOD behavior of TFMs, an area where evidence is scarce. The dataset choice is sensible, and the computational findings on TabFM, LimiX, and Mitra are practically relevant. However, the manuscript's own Figure 3 contains negative shift gaps, which directly contradict the universal-degradation claim in the abstract and Section 3.2. The missing error bars and the absence of a numeric results table also prevent the reader from assessing whether the reported gaps are statistically meaningful. The paper is therefore a valuable contribution in need of substantive revision rather than a conclusive demonstration of its headline claim.","major_comments":[{"comment":"The central claim 'all evaluated TFMs degrade systematically under distribution shift' (Abstract; repeated in §3.2 and §5) is contradicted by the text describing Figure 3: 'Both Mixed and Synthetic include negative gap values, meaning some models in these groups marginally improve under OOD conditions on certain dataset.' Since ΔROC-AUC is defined in §3.1 as ID minus OOD, a negative value means OOD performance exceeds ID performance. This also conflicts with the statement in §3.2 that 'no model achieving OOD parity with its ID performance across any dataset.' The universal quantifier in the headline conclusion is therefore falsified by the paper's own evidence. The claim must be qualified to 'most models on most datasets' or the figures/tables must be reconciled with the text.","section":"§3.2 and Figure 3"},{"comment":"The paper reports that 'All experiments are repeated across five random seeds. We report mean and standard deviation across seeds,' but Figures 1–3 show only point estimates and no error bars, and no numeric table of per-model per-dataset means and standard deviations is provided. Without a measure of seed-to-seed variability, the reader cannot tell whether the observed shift gaps (0.003–0.060) are larger than the noise in the evaluation. Since the paper's main claim is that TFMs systematically degrade, the authors should provide a supplementary table of all reported values and, ideally, a paired significance test (e.g., across seeds) comparing ID and OOD ROC-AUC for each model-dataset combination.","section":"§3.1 and §3.2"},{"comment":"There is an internal inconsistency in the shift-type conclusions. Section 3.2 reports that Childhood Lead (socioeconomic shift) showed the largest degradation, HELOC (label shift) showed a more moderate gap, and Voting (geographic shift) the smallest; yet it concludes that 'label shift is the most relevant shift type for TFMs.' The Discussion repeats this ordering and again associates it with label shift. In the reported data, label shift is not the largest source of degradation. The authors need to correct this attribution, or explain how the TableShift finding they cite supports a label-shift conclusion despite their own ordering.","section":"§3.2 and §4"},{"comment":"The scalability discussion states that 'Mitra imposed a hard limit of 10,000 training rows due to quadratic memory growth, a constraint that affected three of the five datasets considered in this study.' The study considers only three datasets (Table 1). Moreover, Mitra results are reported for Voting, which has 37,548 training rows, so it is unclear how the hard limit was handled. Please clarify whether subsampling or chunking was used for Mitra on Voting and correct the dataset count.","section":"§4"}],"minor_comments":[{"comment":"The scatter plots use different axis ranges across panels, which can exaggerate or minimize visual gaps. Consider using common scales or explicitly noting the ranges in the caption.","section":"§3.2 / Figure 2"},{"comment":"The definition of the shift gap could be stated more clearly: specify whether ΔROC-AUC = ID − OOD or OOD − ID. The current text says 'difference between ID and OOD performance' without a formula; the sign convention matters for interpreting Figure 3.","section":"§3.1"},{"comment":"Model names are inconsistently formatted: 'TabPFNv2' vs 'TabPFN v2', 'Limix' vs 'LimiX', 'TabICLv2' vs 'TabICL v2'. Please standardize.","section":"Throughout"},{"comment":"The claim that 'the relationship between ID and OOD performance documented by Gardner et al. [4] extends to TFMs' is supported only by visual inspection of Figure 2. A correlation coefficient or regression fit would make this claim quantitative.","section":"§4"},{"comment":"The manuscript does not provide a data or code availability statement. Given that the paper is an empirical benchmark, making the evaluation scripts and aggregated results available would substantially strengthen reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely and useful question, but the internal mismatch between the universal claim and the reported negative shift gaps is a load-bearing issue that must be fixed before publication. I would also ask the authors to make the per-seed results available, since the current figures cannot support claims about statistical significance. The paper is not fatally flawed, but it needs a carefully qualified conclusion and a corrected interpretation of the shift-type ordering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this one. First, the paper is genuinely useful as a first systematic look at nine tabular foundation models under the TableShift distribution-shift splits — that comparison did not exist, and the authors ran it with a reasonable setup. Second, the central claim as written is false, and the paper's own figures and text show it. The abstract and conclusions say all TFMs degrade systematically under shift. But §3.2 admits that the Synthetic and Mixed pre-training groups contain negative shift gaps, i.e., at least one model–dataset pair improves on OOD. That single admission breaks the universal quantifier, and it also makes the Fig. 2 statement that no model achieves OOD parity impossible. This is not a subtle representativeness issue; it is an internal contradiction in the reported evidence.\n\nWhat the paper does well: the benchmark design is sound, the model coverage is broad (TabPFN v2–v3, TabICL v1/v2, Mitra, LimiX, TabFM), and the per-dataset ID/OOD plots are informative. The finding that degradation is real but varies by shift type is worth having, and the scalability observations are plausible engineering constraints.\n\nWhere it gets soft, in order of severity: (1) the universal-degradation claim needs to become a 'most models on most datasets' claim, and the authors should either report the negative-gap cases explicitly or reconcile Fig. 3 with Fig. 2; (2) the shift-type conclusion is muddled — Childhood Lead is described as the largest gap, yet the paper concludes label shift is the most relevant, which is not supported by their own ordering; (3) error bars are promised but no standard deviations or numeric table appear; (4) TabFM and LimiX are dropped on Voting without a clear statement of what that does to the aggregate; (5) no classical baselines are run in the same setup, so the 'extends to TFMs' comparison relies on external TableShift numbers; (6) no code or data release; (7) a small but telling slip in §4: the text says 'five datasets' when only three are used.\n\nThe reader's concern about default configurations is minor here. Using each model's documented defaults is a defensible design choice for an initial benchmark, and the paper does not overclaim absolute numbers.\n\nWho should read it: anyone working on tabular model robustness or deciding whether to deploy TFMs in high-stakes settings. The comparison fills a real gap, but the current write-up is not definitive. I would send it to peer review — the result is worth refereeing and the required fixes are local: reword the claims, add the numbers, and fix the shift-type ordering.","headline":"Useful first TFM-vs-TableShift benchmark, but the universal 'all models degrade' claim is contradicted by the paper's own negative shift gaps; the result is worth publishing after a local revision.","tokens_in":8518,"tokens_out":4464,"would_cite":false,"duration_ms":39007,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"All evaluated tabular foundation models degrade under distribution shift","keywords":["tabular foundation models","distribution shift","out-of-distribution","TabPFN","ROC-AUC","TableShift","pre-training strategies","scalability"],"falsifier":"For any of the nine models, re-running the evaluation with the same TableShift splits but finding an OOD ROC-AUC equal to or greater than the ID ROC-AUC (shift gap ≤ 0) would contradict the paper's universal-degradation claim. More narrowly, evaluating a real-data-pretrained TFM on a new socioeconomic shift dataset and observing no drop could break the claim that shift type, not pre-training, dominates.","tokens_in":7673,"feed_emoji":"📉","tokens_out":3817,"duration_ms":32949,"temperature":0.7,"pith_summary":"Tabular foundation models (TFMs) — large models that predict on tables by reading labeled rows as context — appear to hold no built-in protection against distribution shift. The paper tests nine TFMs on three real-world datasets engineered around label, socioeconomic, and geographic shifts, measuring in-distribution versus out-of-distribution ROC-AUC. Every model loses performance when the test distribution changes, with shift gaps between 0.003 and 0.060 depending on the shift type. Pre-training on real-world data lifts absolute accuracy but does not shrink the gap, meaning the advantage comes from stronger in-distribution fitting, not robustness. The findings matter for high-stakes deployment, where benchmarks that ignore shift overstate what TFMs will deliver.","feed_headline":"Every tabular foundation model tested degrades under data shift","feed_subtitle":"Nine models, three shift types: pre-training strategy does not shield against distribution shift, a warning for high-stakes deployment.","key_machinery":"The evaluation rests on the TableShift ID/OOD split protocol: each dataset is divided into a training distribution and a held-out domain (label shift for HELOC, socioeconomic shift for Childhood Lead, geographic shift for Voting). The load-bearing measure is the shift gap (ΔROC-AUC), the difference between in-distribution and out-of-distribution ROC-AUC, which the authors use to separate absolute performance from robustness. The comparison across pre-training strategies (real-world, synthetic, mixed) is what allows the paper to attribute degradation to shift type rather than to a particular training recipe.","core_discovery":"The paper's central claim is that TFMs lack inherent OOD robustness: despite differences in architecture and pre-training corpus, all nine models exhibit consistently lower ROC-AUC on OOD test splits than on ID test splits, on all three datasets. The relationship between ID and OOD performance previously documented for classical tabular models holds for TFMs — higher ID performance maps to proportionally higher OOD performance, with no model escaping the degradation line. Pre-training corpus type does not determine robustness: real-data-pretrained models reach higher absolute OOD scores but display shift gaps comparable to synthetic-prior models. Shift type is the dominant factor, with socio","pith_inferences":["One testable extension is to fine-tune TFMs on a small slice of OOD data and check whether shift gaps shrink; if they do, the lack of robustness may be a limitation of the frozen in-context protocol rather than of TFMs as a class.","The results suggest that OOD-aware data augmentation during pre-training — e.g., simulating label or covariate shifts in the synthetic priors — could be a more direct route to robustness than simply adding more real-world data.","Because only three shift types and three datasets were studied, the claim of 'systematic' degradation is provisional; re-running the protocol on domain shifts like time-based drift or adversarial shifts would test how far the conclusion extends."],"forward_implications":["If TFMs do not possess inherent OOD robustness, then ID benchmark results should not be read as deployment guarantees; ID-improving techniques may not reduce shift-induced errors.","Because shift type, not pre-training strategy, drives degradation, deployment should begin by characterising which type of shift (label, socioeconomic, geographic/covariate) is most likely in the target setting.","Mitra's unusual sensitivity to label shift suggests that mixed SCM/tree priors can be ill-suited to changes in label prevalence, so such models need extra scrutiny in clinical or financial settings with different base rates.","The scalability gap means high-performing TFMs can require memory beyond standard deployment hardware; evaluation suites that run on small datasets will systematically hide this constraint.","The proportional ID-to-OOD relationship implies that among TFMs, the best ID model will usually also be the best OOD model, but the absolute gap still requires measurement."],"fun_headline_variants":["All 9 tabular foundation models fail OOD shift test","Tabular foundation models don't shield from distribution shift","No escape: TFMs degrade under shift regardless of pretraining","Shift type beats pretraining in tabular model OOD robustness","High-performing tabular models demand impractical compute"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The three TableShift ID/OOD splits are representative of the distribution shifts TFMs will actually face, and using each model's default inference configuration gives a fair comparison across models.","fun_headline_variants_meta":{"raw":{"variants":["All 9 tabular foundation models fail OOD shift test","Tabular foundation models don't shield from distribution shift","No escape: TFMs degrade under shift regardless of pretraining","Shift type beats pretraining in tabular model OOD robustness","High-performing tabular models demand impractical compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3039,"prompt_tokens":790,"completion_tokens":2249,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2169}},"tokens_in":534,"tokens_out":2249,"duration_ms":13862,"temperature":1.0,"reasoning_tokens":2169,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T00:51:43.456849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For any of the nine models, re-running the evaluation with the same TableShift splits but finding an OOD ROC-AUC equal to or greater than the ID ROC-AUC (shift gap ≤ 0) would contradict the paper's universal-degradation claim. More narrowly, evaluating a real-data-pretrained TFM on a new socioeconomic shift dataset and observing no drop could break the claim that shift type, not pre-training, dominates.","supporting_citations":[],"review_version":1}