Pith. sign in

REVIEW 4 major objections 5 minor 18 references

All evaluated tabular foundation models degrade under distribution shift

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

All nine tested tabular foundation models degrade under distribution shift, and real-world pretraining provides no robustness advantage over synthetic pretraining.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection Useful first TFM-vs-TableShift benchmark, but the universal 'all models degrade' claim is contradicted by the paper's own negative shift gaps; the result is worth publishing after a local revision. the 4 major comments →

arxiv 2607.26000 v1 pith:3UOJSXAI submitted 2026-07-28 cs.LG cs.AI

Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models

Malena Loza , David Chushig-Muzo , Eva Milara , Luis Bote-Curiel , Luis Estrada-Petrocelli , Felipe Grijalva This is my paper
classification cs.LG cs.AI
keywords tabular foundation modelsdistribution shiftout-of-distributionTabPFNROC-AUCTableShiftpre-training strategiesscalability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Tabular foundation models (TFMs) — large models that predict on tables by reading labeled rows as context — appear to hold no built-in protection against distribution shift. The paper tests nine TFMs on three real-world datasets engineered around label, socioeconomic, and geographic shifts, measuring in-distribution versus out-of-distribution ROC-AUC. Every model loses performance when the test distribution changes, with shift gaps between 0.003 and 0.060 depending on the shift type. Pre-training on real-world data lifts absolute accuracy but does not shrink the gap, meaning the advantage comes from stronger in-distribution fitting, not robustness. The findings matter for high-stakes deployment, where benchmarks that ignore shift overstate what TFMs will deliver.

Core claim

The paper's central claim is that TFMs lack inherent OOD robustness: despite differences in architecture and pre-training corpus, all nine models exhibit consistently lower ROC-AUC on OOD test splits than on ID test splits, on all three datasets. The relationship between ID and OOD performance previously documented for classical tabular models holds for TFMs — higher ID performance maps to proportionally higher OOD performance, with no model escaping the degradation line. Pre-training corpus type does not determine robustness: real-data-pretrained models reach higher absolute OOD scores but display shift gaps comparable to synthetic-prior models. Shift type is the dominant factor, with socio

What carries the argument

The evaluation rests on the TableShift ID/OOD split protocol: each dataset is divided into a training distribution and a held-out domain (label shift for HELOC, socioeconomic shift for Childhood Lead, geographic shift for Voting). The load-bearing measure is the shift gap (ΔROC-AUC), the difference between in-distribution and out-of-distribution ROC-AUC, which the authors use to separate absolute performance from robustness. The comparison across pre-training strategies (real-world, synthetic, mixed) is what allows the paper to attribute degradation to shift type rather than to a particular training recipe.

Load-bearing premise

The three TableShift ID/OOD splits are representative of the distribution shifts TFMs will actually face, and using each model's default inference configuration gives a fair comparison across models.

What would settle it

For any of the nine models, re-running the evaluation with the same TableShift splits but finding an OOD ROC-AUC equal to or greater than the ID ROC-AUC (shift gap ≤ 0) would contradict the paper's universal-degradation claim. More narrowly, evaluating a real-data-pretrained TFM on a new socioeconomic shift dataset and observing no drop could break the claim that shift type, not pre-training, dominates.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If TFMs do not possess inherent OOD robustness, then ID benchmark results should not be read as deployment guarantees; ID-improving techniques may not reduce shift-induced errors.
  • Because shift type, not pre-training strategy, drives degradation, deployment should begin by characterising which type of shift (label, socioeconomic, geographic/covariate) is most likely in the target setting.
  • Mitra's unusual sensitivity to label shift suggests that mixed SCM/tree priors can be ill-suited to changes in label prevalence, so such models need extra scrutiny in clinical or financial settings with different base rates.
  • The scalability gap means high-performing TFMs can require memory beyond standard deployment hardware; evaluation suites that run on small datasets will systematically hide this constraint.
  • The proportional ID-to-OOD relationship implies that among TFMs, the best ID model will usually also be the best OOD model, but the absolute gap still requires measurement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to fine-tune TFMs on a small slice of OOD data and check whether shift gaps shrink; if they do, the lack of robustness may be a limitation of the frozen in-context protocol rather than of TFMs as a class.
  • The results suggest that OOD-aware data augmentation during pre-training — e.g., simulating label or covariate shifts in the synthetic priors — could be a more direct route to robustness than simply adding more real-world data.
  • Because only three shift types and three datasets were studied, the claim of 'systematic' degradation is provisional; re-running the protocol on domain shifts like time-based drift or adversarial shifts would test how far the conclusion extends.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical benchmark of nine tabular foundation models (TabPFN variants, TabICL variants, Mitra, LimiX, TabFM) on three TableShift datasets covering label, socioeconomic, and geographic distribution shifts. The authors report ID and OOD ROC-AUC for each model-dataset pair, define a shift gap, and claim that all evaluated TFMs degrade systematically under distribution shift, that real-world pre-training does not confer OOD robustness, that ID-OOD performance relationships known for classical models extend to TFMs, and that scalability limits several high-performing models. The results are based on five random seeds and default inference configurations.

Significance. If the central claim were fully supported, this would be a useful early benchmark for OOD behavior of TFMs, an area where evidence is scarce. The dataset choice is sensible, and the computational findings on TabFM, LimiX, and Mitra are practically relevant. However, the manuscript's own Figure 3 contains negative shift gaps, which directly contradict the universal-degradation claim in the abstract and Section 3.2. The missing error bars and the absence of a numeric results table also prevent the reader from assessing whether the reported gaps are statistically meaningful. The paper is therefore a valuable contribution in need of substantive revision rather than a conclusive demonstration of its headline claim.

major comments (4)
  1. [§3.2 and Figure 3] The central claim 'all evaluated TFMs degrade systematically under distribution shift' (Abstract; repeated in §3.2 and §5) is contradicted by the text describing Figure 3: 'Both Mixed and Synthetic include negative gap values, meaning some models in these groups marginally improve under OOD conditions on certain dataset.' Since ΔROC-AUC is defined in §3.1 as ID minus OOD, a negative value means OOD performance exceeds ID performance. This also conflicts with the statement in §3.2 that 'no model achieving OOD parity with its ID performance across any dataset.' The universal quantifier in the headline conclusion is therefore falsified by the paper's own evidence. The claim must be qualified to 'most models on most datasets' or the figures/tables must be reconciled with the text.
  2. [§3.1 and §3.2] The paper reports that 'All experiments are repeated across five random seeds. We report mean and standard deviation across seeds,' but Figures 1–3 show only point estimates and no error bars, and no numeric table of per-model per-dataset means and standard deviations is provided. Without a measure of seed-to-seed variability, the reader cannot tell whether the observed shift gaps (0.003–0.060) are larger than the noise in the evaluation. Since the paper's main claim is that TFMs systematically degrade, the authors should provide a supplementary table of all reported values and, ideally, a paired significance test (e.g., across seeds) comparing ID and OOD ROC-AUC for each model-dataset combination.
  3. [§3.2 and §4] There is an internal inconsistency in the shift-type conclusions. Section 3.2 reports that Childhood Lead (socioeconomic shift) showed the largest degradation, HELOC (label shift) showed a more moderate gap, and Voting (geographic shift) the smallest; yet it concludes that 'label shift is the most relevant shift type for TFMs.' The Discussion repeats this ordering and again associates it with label shift. In the reported data, label shift is not the largest source of degradation. The authors need to correct this attribution, or explain how the TableShift finding they cite supports a label-shift conclusion despite their own ordering.
  4. [§4] The scalability discussion states that 'Mitra imposed a hard limit of 10,000 training rows due to quadratic memory growth, a constraint that affected three of the five datasets considered in this study.' The study considers only three datasets (Table 1). Moreover, Mitra results are reported for Voting, which has 37,548 training rows, so it is unclear how the hard limit was handled. Please clarify whether subsampling or chunking was used for Mitra on Voting and correct the dataset count.
minor comments (5)
  1. [§3.2 / Figure 2] The scatter plots use different axis ranges across panels, which can exaggerate or minimize visual gaps. Consider using common scales or explicitly noting the ranges in the caption.
  2. [§3.1] The definition of the shift gap could be stated more clearly: specify whether ΔROC-AUC = ID − OOD or OOD − ID. The current text says 'difference between ID and OOD performance' without a formula; the sign convention matters for interpreting Figure 3.
  3. [Throughout] Model names are inconsistently formatted: 'TabPFNv2' vs 'TabPFN v2', 'Limix' vs 'LimiX', 'TabICLv2' vs 'TabICL v2'. Please standardize.
  4. [§4] The claim that 'the relationship between ID and OOD performance documented by Gardner et al. [4] extends to TFMs' is supported only by visual inspection of Figure 2. A correlation coefficient or regression fit would make this claim quantitative.
  5. [General] The manuscript does not provide a data or code availability statement. Given that the paper is an empirical benchmark, making the evaluation scripts and aggregated results available would substantially strengthen reproducibility.

Circularity Check

0 steps flagged

No circularity found: the study is a purely empirical benchmark with no fitted-input-as-prediction, self-citation chain, or construction-level dependence on its own outputs.

full rationale

This paper is an experimental evaluation, not a derivation. The central claim—that TFMs degrade under distribution shift—is supported by measured ID vs. OOD ROC-AUC comparisons on external TableShift splits (HELOC, Childhood Lead, Voting). No parameter is fitted to the outcome and then renamed as a prediction; no target quantity is defined in terms of the claimed result. The models are used in their default configurations, and the datasets and shift types are taken from an external benchmark [4]. The only self-citations (refs. 3 and 8, involving one of the present authors) concern unrelated prior biomedical applications and are not load-bearing for the OOD conclusions. There is no ansatz imported via self-citation and no uniqueness theorem invoked. The reader's initial score of 0 is appropriate. One non-circularity concern should be noted for correctness, not circularity: the text in §3.2 says that 'Both Mixed and Synthetic include negative gap values, meaning some models in these groups marginally improve under OOD conditions on certain dataset,' while the Abstract and §5 claim that 'all evaluated TFMs degrade systematically.' Negative shift gaps contradict the universal quantifier in that claim, and the ordering of largest degradation on Childhood Lead versus the conclusion that 'label shift is the most relevant shift type' is also internally strained. These are evidence–claim consistency problems, not circularity: the reported measurements are independent of the conclusion and could in principle falsify it. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No free parameters are fitted; the paper is an empirical benchmark. It rests on domain assumptions about the representativeness of TableShift splits and the fairness of default model configurations.

axioms (3)
  • domain assumption TableShift ID/OOD splits define valid distribution shifts representative of deployment shifts.
    The conclusions about OOD robustness depend entirely on these splits being representative; §2.1.
  • domain assumption Default inference configuration without fine-tuning is a fair evaluation of TFM capability.
    §3.1 states all TFMs are used in default configurations, so observed degradation is attributed to the models rather than to configuration choices.
  • domain assumption ROC-AUC is an appropriate evaluation metric for these imbalanced classification tasks.
    §3.1 justifies ROC-AUC due to class imbalance, but no alternative metric is reported.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models." pith.science (2026). https://pith.science/paper/3UOJSXAI

@misc{pith2026260726000,
  author       = {Pith},
  title        = {Pith review of: Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3UOJSXAI}},
  note         = {Machine review of arXiv:2607.26000}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Tabular Foundation Models (TFMs) have emerged as novel approaches for tabular predictive tasks, demonstrating competitive predictive performance to ensemble tree-based models. Most TFMs are trained and evaluated on independent and identically distributed data, but this assumption changes in real-world scenarios due to distribution shifts, which compromise the robustness of models. Limited research has been conducted of TFMs under distribution shifts. We present an empirical evaluation of Out-Of-Distribution (OOD) performance of nine TFMs, spanning diverse pre-training strategies and architectures: TabPFNv2, TabPFNv2.5, TabPFNv2.6, TabPFNv3, TabICL, TabICLv2, Mitra, LimiX and TabFM. Three real-world datasets from the TableShift study were considered (HELOC, Voting, Childhood Lead), covering label, socioeconomic, and geographic shift types. Our results show that all evaluated TFMs degrade systematically under distribution shift regardless of pre-training strategy, with shift gaps ranging from 0.003 to 0.060 depending on shift type. The relationship between in-distribution and OOD predictive performance documented for classical tabular models extends into TFMs. We also identified a scalability gap, as high-performing models demand significant memory and computational resources beyond what standard deployment infrastructure can support. This study extends existing benchmarks for OOD in tabular data, providing evidence to support their adoption in high-stakes domains characterized by structural distribution shifts.

Figures

Figures reproduced from arXiv: 2607.26000 by David Chushig-Muzo, Eva Milara, Felipe Grijalva, Luis Bote-Curiel, Luis Estrada-Petrocelli, Malena Loza.

Figure 1
Figure 1. Figure 1: In-distribution and out-of-distribution results for all tabular foundation [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: In-distribution versus out-of-distribution ROC-AUC scatter plots per [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pre-training corpus effect on out-of-distribution robustness. Left panel: [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 2 linked inside Pith

  1. [1]

    In: Pro- ceedings of the AAAI conference on artificial intelligence

    Arik, S.Ö., Pfister, T.: Tabnet: Attentive interpretable tabular learning. In: Pro- ceedings of the AAAI conference on artificial intelligence. vol. 35, pp. 6679–6687 (2021)

    2021

  2. [2]

    Behre, O.: In-context forecasting in supply chains: Evaluating the promise and limits of tabular foundation models (2026) 10 Loza-Casa M. et al

    2026

  3. [3]

    Applied Sciences 14(21), 9870 (2024)

    Chushig-Muzo, D., Calero-Díaz, H., Fabelo, H., Årsand, E., van Dijk, P.R., Soguero-Ruiz, C.: Characterizing the impact of physical activity on patients with type 1 diabetes using statistical and machine learning models. Applied Sciences 14(21), 9870 (2024)

    2024

  4. [4]

    Advances in Neural Information Processing Systems36, 53385–53432 (2023)

    Gardner, J., Popovic, Z., Schmidt, L.: Benchmarking distribution shift in tabu- lar data with tableshift. Advances in Neural Information Processing Systems36, 53385–53432 (2023)

    2023

  5. [5]

    arXiv preprint arXiv:2605.13986 (2026)

    Grinsztajn, L., Flöge, K., Key, O., Birkel, F., Jund, P., Roof, B., Manium, M., Hoo, S.B., Bühler, M., Garg, A., et al.: Tabpfn-3: Technical report. arXiv preprint arXiv:2605.13986 (2026)

    Pith/arXiv arXiv 2026

  6. [6]

    Nature637(8045), 319–326 (2025)

    Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S.B., Schirrmeister, R.T., Hutter, F.: Accurate predictions on small data with a tabular foundation model. Nature637(8045), 319–326 (2025)

    2025

  7. [7]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2026)

    Jiang, J.P., Liu, S.Y., Cai, H.R., Zhou, Q.L., Ye, H.J.: Representation learning for tabular data: A comprehensive survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2026)

    2026

  8. [8]

    Engineering Applications of Artificial Intelligence144, 110142 (2025)

    Lara-Abelenda, F.J., Chushig-Muzo, D., Wägner, A.M., Tayefi, M., Soguero-Ruiz, C.: Interpretable and multimodal fusion methodology to predict severe hypo- glycemia in adults with type 1 diabetes. Engineering Applications of Artificial Intelligence144, 110142 (2025)

    2025

  9. [9]

    Neural Networks p

    Lee, K.: Multitabpfn: Codebook-based extensions of tabpfn for high-class-count tabular classification. Neural Networks p. 108932 (2026)

    2026

  10. [10]

    arXiv preprint arXiv:2410.18164 (2024)

    Ma, J., Thomas, V., Hosseinzadeh, R., Labach, A., Kamkari, H., Cresswell, J.C., Golestan, K., Yu, G., Caterini, A.L., Volkovs, M.: Tabdpt: Scaling tabular foun- dation models on real data. arXiv preprint arXiv:2410.18164 (2024)

    arXiv 2024

  11. [11]

    arXiv preprint arXiv:2602.11139 (2026)

    Qu, J., Holzmüller, D., Varoquaux, G., Morvan, M.L.: Tabiclv2: A better, faster, scalable, and open tabular foundation model. arXiv preprint arXiv:2602.11139 (2026)

    arXiv 2026

  12. [12]

    Fu- ture Generation Computer Systems166, 107707 (2025)

    Ruiz-Villafranca, S., Roldán-Gómez, J., Carrillo-Mondejar, J., Martinez, J.L., Gañán, C.H.: Wfe-tab: Overcoming limitations of tabpfn in iiot-mec environments with a weighted fusion ensemble-tabpfn model for improved ids performance. Fu- ture Generation Computer Systems166, 107707 (2025)

    2025

  13. [13]

    ruiz-villafranca et al

    Ruiz-Villafranca, S., Roldán-Gómez, J., Gómez, J.M.C., Carrillo-Mondéjar, J., Martinez, J.L.: A tabpfn-based intrusion detection system for the industrial in- ternet of things: S. ruiz-villafranca et al. The Journal of Supercomputing80(14), 20080–20117 (2024)

    2024

  14. [14]

    arXiv preprint arXiv:2106.01342 (2021)

    Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C.B., Goldstein, T.: Saint: Improved neural networks for tabular data via row attention and contrastive pre- training. arXiv preprint arXiv:2106.01342 (2021)

    Pith/arXiv arXiv 2021

  15. [15]

    ACM Computing Surveys (2026)

    Somvanshi, S., Das, S., Javed, S., Antariksa, G., Hossain, A.: A survey on tabular data: from tree-based methods to tabular deep learning. ACM Computing Surveys (2026)

    2026

  16. [16]

    International Journal of Computer Vision132(12), 5635–5662 (2024)

    Yang, J., Zhou, K., Li, Y., Liu, Z.: Generalized out-of-distribution detection: A survey. International Journal of Computer Vision132(12), 5635–5662 (2024)

    2024

  17. [17]

    arXiv preprint arXiv:2509.03505 (2025)

    Zhang, X., Ren, G., Yu, H., Yuan, H., Wang, H., Li, J., Wu, J., Mo, L., Mao, L., Hao, M., et al.: Limix: Unleashing structured-data modeling capability for generalist intelligence. arXiv preprint arXiv:2509.03505 (2025)

    arXiv 2025

  18. [18]

    Advances in neural information processing systems38, 15795–15840 (2026)

    Zhang, X., Maddix Robinson, D., Yin, J., Erickson, N., Ansari, A.F., Han, B., Zhang, S., Akoglu, L., Faloutsos, C., Mahoney, M., et al.: Mitra: Mixed synthetic priors for enhancing tabular foundation models. Advances in neural information processing systems38, 15795–15840 (2026)

    2026

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.