Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An adversarially pre-trained transformer matches boosted trees on small tabular datasets in under a second per dataset, with no per-dataset training.

desk verdict Legit TabPFN extension with two real novelties (adversarial data agents, mixture block), but the headline gain over TabPFN is small and statistically under-supported. read the letter →

arxiv 2502.04573 v2 pith:372H6GQF submitted 2025-02-06 cs.LG cs.AI

Yulun Wu , Doron L. Bergman This is my paper · ORCID
classification cs.LGcs.AI
keywords zero-shotmeta-learningtabularpredictionadversarialpre-trainingprior-datafittednetworksmixtureblocktransformersyntheticdataclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that a transformer pre-trained only on synthetic data can perform zero-shot prediction on small real-world tabular datasets, matching or beating gradient-boosted decision trees without any per-dataset model fitting. The authors claim their Adversarially Pre-trained Transformer (APT) reaches state-of-the-art ROC-AUC on all 35 small OpenML-CC18 classification tasks, including datasets with many classes, categorical features, and missing values that prior zero-shot learners had to exclude. They also claim APT improves over TabPFN on tabular regression. If correct, this makes zero-shot meta-learning a practical alternative to traditional one-model-per-dataset pipelines for small tabular problems, with inference in under a second.

What carries the argument

Two components carry the argument. First, adversarial data agents: MLP-based synthetic data generators, a fraction of the generator pool, that receive gradient updates to increase the meta-learner's prediction loss, with a soft-discretization trick (extending ranking discretization with a temperature-controlled continuous relaxation, inspired by Gumbel-Softmax and Concrete distributions) to preserve end-to-end gradient flow through categorical features, and periodic resets to prevent mode collapse. Second, the mixture block: a classification output head that, for each test point, computes attention weights over training embeddings, uses Concrete-distribution-sampled binary gates to sparsify which training points contribute, and then scatter-sums gated probability mass by class index to produce predictions, eliminating the need for a fixed-size dense output layer.

What would settle it

Train APT on the exact same synthetic generator mix but with all adversarial agents replaced by ordinary random generators while keeping the mixture block, and compare zero-shot ROC-AUC on the 35 small OpenML-CC18 datasets; if the gap to the adversarial version vanishes, the claimed benefit of adversarial pre-training would be falsified on the paper's own evaluation pool.

Watch

Extended reading notes

Core claim

The central claim is that adversarial pre-training on synthetic data substantially improves the zero-shot tabular prediction capability of Prior-Data Fitted Networks, and that a novel mixture block output head removes the class-size limitation of TabPFN. APT is pre-trained once, offline, using a mix of ordinary random MLP generators and adversarial data agents that are updated by gradient ascent on the prediction loss, forcing the model to handle increasingly difficult synthetic datasets. The paper reports that APT achieves a mean ROC-AUC of 0.921 over 35 small OpenML-CC18 datasets with a mean rank of 3.86, slightly ahead of CatBoost (4.03) and TabPFN (4.57), while keeping average runtime at 0.90 seconds on an H100 GPU. On regression, APT lowers mean MSE on OpenML-CTR23 from 0.412 to 0.344 on small datasets, both in the classification and regression ablation settings. The mixture block, which queries training embeddings with Concrete-distribution gates and scatter-sums by class label, allows the model to handle an arbitrary number of classes and accelerates pre-training, reaching 0.90 AUC in 11,780 gradient steps versus much longer for the dense-head baseline.

Load-bearing premise

The synthetic data distribution, even after adversarial expansion, faithfully covers the range of patterns found in real-world tabular data, so that a model pre-trained only on synthetic data transfers to unseen real datasets.

Editorial extensions

If this is right

  • APT offers a drop-in zero-shot alternative to per-dataset model training on small tabular problems, producing predictions in under a second on a GPU with no hyperparameter tuning or backpropagation at inference time.
  • The class-size-freedom of the mixture block means a deployed zero-shot classifier need not know the label cardinality in advance, removing a key practical obstacle for TabPFN-based systems.
  • Adversarial pre-training appears to specifically close the gap on datasets with missing values and categorical features, the acknowledged weak spots of the prior TabPFN model.
  • If the regression gains hold, zero-shot meta-learning becomes competitive for small regression tasks as well, where prior PFN approaches lagged noticeably behind tree-based methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The described adversarial data agents may serve as a general-purpose data augmentation strategy for any in-context learner, not just tabular transformers, because they probe the model's decision boundary and generate hard but learnable examples.
  • A direct head-to-head of APT against a TabPFN variant trained with the same mixture block but without adversarial agents is the cleanest way to isolate the contribution of adversarial pre-training; the paper's ablation compares full APT to TabPFN-with-mixture, leaving the interaction between the two components partly entangled.
  • The soft-discretization mechanism could be reused as a differentiable way to generate categorical features in other synthetic-data pipelines, since it preserves gradients while staying close to a discrete ranking scheme.
  • One testable prediction of the paper's framing is that increasing the fraction of adversarial agents beyond 12.5% would eventually hurt diversity rather than help, because agents may converge to a narrow set of hard distributions; the reset cadence would then need to scale accordingly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Adversarially Pre-trained Transformer (APT), a zero-shot meta-learner for tabular prediction. APT is pre-trained exclusively on synthetic data, mixing the ordinary sparse-MLP generators of TabPFN with a subset of adversarial data agents that shift their generating distributions to make datasets harder for the model. The authors also introduce a mixture block output head that replaces the fixed-size dense classifier with a scatter-sum over training labels, claiming it removes the class-cardinality restriction of prior PFN models. Experiments are reported on 35 small OpenML-CC18 classification datasets without the filtering used in earlier TabPFN work, and on the OpenML-CTR23 regression suite. The main claims are that APT matches or slightly surpasses state-of-the-art GBDT and TabPFN performance on small classification tasks, improves regression over TabPFN, that adversarial agents increase synthetic-data diversity, and that the mixture block accelerates pre-training.

Significance. If the empirical claims hold, this is a useful step for zero-shot tabular meta-learning: it is, to my knowledge, the first demonstration that adversarial synthetic-data generation can improve PFN transfer on broad tabular benchmarks, and the mixture block is a sensible architectural response to the class-cardinality limitation. The paper has real strengths: the evaluation suite is not filtered by class count, missing values, or categorical features; the released checkpoint is evaluated without ad hoc checkpoint selection; the ablation study separates the contributions of adversarial pre-training and the mixture block; and the authors state limitations about large datasets and quadratic complexity. The main risk is that the headline aggregate gains are small and not supported by paired significance testing, and that the TabPFN baseline used for comparison is not the standard released model. The central claim is plausible but statistically under-supported as written.

major comments (5)
  1. [§3.1, Table 1] The main 'state-of-the-art' claim rests on aggregate ROC-AUC differences of 0.003–0.008 (APT 0.921 vs CatBoost 0.918 vs TabPFN 0.913), while the reported 'std. of mean' column measures variation across splits, not across datasets or across pre-training seeds. No paired significance test across the 35 datasets is reported, and Table 5 shows clear losses for APT on cnae-9 (0.901 vs 0.991 for CatBoost), semeion (0.980 vs 0.999), and car (0.997 vs 1.000). I ask for paired tests (e.g., Wilcoxon signed-rank on per-dataset mean AUC), confidence intervals over multiple pre-training seeds, and a discussion of the failure cases before the aggregate claim is accepted.
  2. [§3, Hyperparameters paragraph] The TabPFN comparator is not clearly the standard released model. The text says TabPFN's hyperparameters are 'directly inherited from their released checkpoint, only changing the maximum number of classes from 10 to 26'. Since the maximum class count is a pre-training hyperparameter, this either requires retraining or continued pre-training; the manuscript does not specify which, nor does it identify the resulting checkpoint. The paper should compare against both the officially released TabPFN checkpoint and a retrained 26-class version, with training details, so that the 'enhances TabPFN' claim is interpretable.
  3. [§3.4, Table 4] The regression comparison is against a TabPFN model trained only on classification problems, as the authors concede. The large MSE improvement (0.412 to 0.344 on small datasets) may therefore reflect a weak baseline rather than a genuine regression advance. The proper control is an ordinary (non-adversarial) TabPFN pre-trained under the same regression objective and data-generating setup; without it, the claim that adversarial pre-training 'boosted' regression performance is overstated.
  4. [§2.2, Figure 2] The mixture block is central to the paper's generalizability claim, but the description of the scatter-sum output head is incomplete. The text says gated probabilities from the same class are added together to yield final predictions and calls them class probabilities, yet it does not state whether the class-wise sums are normalized after aggregation. If a normalization step exists, it should be specified; if not, the authors should explain how scores for classes with very different training-set frequencies are comparable, since valid probabilities are required for the claimed arbitrary-class-count behavior.
  5. [§2.1, Eq. (1)] Equation (1) writes the adversarial agent's objective as L(gη) = E log qθ(y|X), which is a log-likelihood, not a loss to be maximized by an agent that wants to increase prediction loss. The following paragraph invokes a sign flip between generator and discriminator gradients, but the direction of the agent's update is not unambiguous from the equation alone. Please define explicitly whether the agent maximizes −log qθ, or equivalently whether Eq. (1) is the model's log-likelihood and the agent's gradient is the negative of the discriminator's gradient on the negative log-likelihood.
minor comments (5)
  1. [§3, adversarial agent hyperparameters] The adversarial-agent hyperparameters (fraction 12.5%, learning rate 10^−1, weight decay 10^−5, temperature 10^−2, reset interval 2,000 steps) are reported without stating how they were selected. If any of these were chosen using the OpenML-CC18 evaluation pool, that should be disclosed; ideally a synthetic-only or separate validation suite would be used.
  2. [§3.2, Figure 3 and KL values] The diversity claim is supported by a KL divergence of 0.134 ± 0.141 between two ordinary-generator collections and 0.813 ± 0.072 between ordinary and adversarial collections; the very large standard deviation in the former makes the visual claim of near-identical ordinary distributions hard to interpret, and I suggest reporting a more direct overlap or divergence with confidence intervals.
  3. [§3.1, Figure 4] In the sentence 'has closened the gap', 'closened' is a typo; it should be 'narrowed'.
  4. [Table 2] The entries '1 ±0' should read '1.000 ± 0.000' or '1 ± 0' with a consistent number of decimals, as this formatting makes the near-saturated results look like exact integers.
  5. [§3.3, Figure 5] The learning-curve experiment appears to be a single run without error bars or multiple seeds; this should be stated explicitly in the caption so readers do not infer run-to-run stability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: APT's pre-training uses synthetic data only and its benchmark claims are evaluated on held-out real-world datasets; the diversity analysis is an empirical sanity check, not a load-bearing derivation.

full rationale

The paper's derivation chain is self-contained with respect to its central claims. APT is pre-trained entirely on synthetic data generated by ordinary MLP generators plus adversarial agents, with no real-world data used during pre-training; the main evaluations on OpenML-CC18 classification and CTR23 regression are external held-out benchmarks. The adversarial-agent objective in Eq. (1) is a genuine training mechanism that maximizes the model's prediction loss on generated data, and the claimed performance improvements over TabPFN are empirical comparisons supported by ablations (Tables 3 and 4). The Section 3.2 diversity analysis measures a property of the generated synthetic distribution; although the fact that adversarial agents shift their generating distribution is close to their defining objective, the reported KL-divergence result is a posterior empirical observation rather than a reused input, and it does not by itself establish any of the paper's benchmark claims. No load-bearing self-citation chain appears: TabPFN and PFN citations are external prior work, and the paper does not invoke a uniqueness theorem or provenance-specific mathematical result to force its conclusions. The concerns raised by the skeptic—small aggregate performance margins, a retrained TabPFN baseline with modified maximum class count, absence of paired significance tests, and a possibly under-optimized regression baseline—are statistical and comparability risks, not circularity of the derivation.

Assumptions & free parameters 6 free parameters · 4 assumptions · 2 invented entities

The central claim depends on the synthetic prior family inherited from TabPFN, the novel adversarial training procedure with its hand-set hyperparameters, and the assumption that the mixture block's inductive bias transfers to unseen class counts. No real-world data is used in pre-training, so the benchmark evaluations are held-out. The free parameters listed here are all chosen by the authors and none are accompanied by sensitivity analyses.

free parameters (6)
  • Fraction of adversarial data agents = 12.5%
    Chosen by the authors; no sensitivity analysis or separate validation is reported (Section 3, paragraph 2).
  • Adversarial agent learning rate = 1e-1
    Chosen by the authors for the data-agent updates (Section 3, paragraph 2).
  • Adversarial agent weight decay = 1e-5
    Chosen by the authors for the data-agent updates (Section 3, paragraph 2).
  • Soft-discretization temperature = 1e-2
    Controls the differentiable relaxation in Eq. 2; no sensitivity analysis provided (Section 3, paragraph 2).
  • Data agent reset interval = 2000 gradient steps
    Frequency of re-initializing adversarial agents to avoid mode collapse; chosen by hand (Section 2.1, last paragraph; Section 3).
  • Mixture block Concrete gate temperature = not reported
    The gates in the mixture block are sampled via Concrete distribution; the temperature is not specified in the paper, a free design choice.
assumptions (4)
  • standard math The PFN objective (Eq. 10) approximates Bayesian posterior predictive inference for synthetic priors
    Inherited from Muller et al. (2021) and stated in Appendix A; APT relies on this equivalence without reproving it.
  • domain assumption Sparsified noisy random MLPs form a useful prior family for real-world tabular datasets
    Inherited from TabPFN (Hollmann et al., 2022); the paper extends this prior with adversarial agents but does not validate the prior on real data.
  • ad hoc to paper Adversarial gradient updates on MLP data agents explore challenging but informative distributions rather than collapsing to no-signal noise
    The paper relies on data agent reset to avoid collapse (Section 2.1, last paragraph) but provides no formal guarantee; Section 3.2 reports that with their settings, features remain correlated with responses.
  • ad hoc to paper The mixture block's scatter-sum over training labels yields valid posterior probabilities for arbitrary class counts
    Proposed in Section 2.2; only tested on up to 26 classes and on 4 datasets in Section 3.3, so the 'arbitrary number of classes' claim is an assumption.
invented entities (2)
  • Adversarial synthetic data agents
    purpose: MLP-based generators that shift their data-generating distribution by gradient ascent on the learner's loss to produce more challenging synthetic datasets during pre-training
    The only evidence for their utility is the paper's own ablations and the KL-divergence analysis on synthetic data generated by the agents themselves; no independent reproduction is provided.
  • Mixture block
    purpose: Classification output head that queries training embeddings, gates them via Concrete distribution, and scatter-sums by training label to handle arbitrary class counts
    Its generalizability claim is supported only by the paper's Table 2 (4 datasets) and training-acceleration Figure 5; no external validation or analysis of failure cases is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer." pith.science (2026). https://pith.science/paper/372H6GQF

@misc{pith2026250204573,
  author       = {Pith},
  title        = {Pith review of: Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/372H6GQF}},
  note         = {Machine review of arXiv:2502.04573}
}
read the original abstract

We present an Adversarially Pre-trained Transformer (APT) that is able to perform zero-shot meta-learning on tabular prediction tasks without pre-training on any real-world dataset, extending on the recent development of Prior-Data Fitted Networks (PFNs) and TabPFN. Specifically, APT is pre-trained with adversarial synthetic data agents, who continue to shift their underlying data generating distribution and deliberately challenge the model with different synthetic datasets. In addition, we propose a mixture block architecture that is able to handle classification tasks with arbitrary number of classes, addressing the class size limitation -- a crucial weakness of prior deep tabular zero-shot learners. In experiments, we show that our framework matches state-of-the-art performance on small classification tasks without filtering on dataset characteristics such as number of classes and number of missing values, while maintaining an average runtime under one second. On common benchmark dataset suites in both classification and regression, we show that adversarial pre-training was able to enhance TabPFN's performance. In our analysis, we demonstrate that the adversarial synthetic data agents were able to generate a more diverse collection of data compared to the ordinary random generator in TabPFN. In addition, we demonstrate that our mixture block neural design has improved generalizability and greatly accelerated pre-training.

Figures

Figures reproduced from arXiv: 2502.04573 by the authors.

Figure 1
Figure 1. The model workflow of Adversarially Pre-trained Transformer (APT). Pre-training is done once, offline, with datasets generated by a mix of random synthetic data generators and adversarial synthetic data agents. The train-test split is randomly sampled for each batch of datasets. After the model is pre-trained and deployed, predictions are done per real-world dataset, online, with one forward pass and no parameter up… view at source ↗
Figure 2
Figure 2. Model architecture and the mixture block. a) X = (X train, Xtest) and y train are embedded on R dmodel using a feature embedding block and linear projection respectively. Then, embeddings for X train and y train are added as h train, embeddings for X test are denoted as h test. Embeddings (h train , h test) are then passed to the transformer blocks with attention towards test embedding h test masked, same as Hollman… view at source ↗
Figure 3
Figure 3. Contour plot of two-dimensional data generated by ordinary data generators and adversarial data agents. Each subplot contains a total of 100,000 data points from 2,000 datasets. Note that subplot (a) and subplot (b) are two independent sets of ordinary generators with no mutual, as each dataset is generated by an independently initialized random sparse neural network. Each dataset in subplot (c) is generated by an a… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: A breakdown of performance by dataset characteristics. The mean of ranks are plotted as orange on datasets with the respec￾tive characteristic, and as blue on datasets without the respective characteristic. From [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: ROC-AUC on the 35 small datasets in OpenML-CC18 evaluated after each of the first 30,000 gradient steps. From [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Discretization of continuous variables. x-axis is the value generated by the data generator, and y-axis is its value after discretization. The soft-discretization approach produces near-categorical features that are differentiable and thus do not disrupt gradient flow.…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SMETA-ZSL:Semantic Meta-Alignment for Zero-Shot Threat Classification

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SMETA-ZSL combines contrastive LLM prototypes, episodic meta-alignment, and Z-score gating to beat prior generalized zero-shot methods by ~10.8 harmonic-mean points under strict open-set inductive conditions on seven ...

Reference graph

Works this paper leans on

89 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Devanbu, P

    Ahmed, T. and Devanbu, P. Few-shot training llms for project-specific code-summarization. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp.\ 1--5, 2022

    2022

  3. [3]

    and Flammarion, N

    Andriushchenko, M. and Flammarion, N. Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33: 0 16048--16059, 2020

    2020

  4. [4]

    Arik, S. \"O . and Pfister, T. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.\ 6679--6687, 2021

    2021

  5. [5]

    G., van Rijn, J

    Bischl, B., Casalicchio, G., Feurer, M., Gijsbers, P., Hutter, F., Lang, M., Mantovani, R. G., van Rijn, J. N., and Vanschoren, J. Openml benchmarking suites. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021

    2021

  6. [6]

    Deep neural networks and tabular data: A survey

    Borisov, V., Leemann, T., Se ler, K., Haug, J., Pawelczyk, M., and Kasneci, G. Deep neural networks and tabular data: A survey. IEEE transactions on neural networks and learning systems, 2022

    2022

  7. [7]

    Language models are realistic tabular data generators

    Borisov, V., Sessler, K., Leemann, T., Pawelczyk, M., and Kasneci, G. Language models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=cEygmQNOeI

    2023

  8. [8]

    Bulatov, A., Kuratov, Y., and Burtsev, M. S. Scaling transformer to 1m tokens and beyond with RMT . CoRR, abs/2304.11062, 2023. doi:10.48550/ARXIV.2304.11062. URL https://doi.org/10.48550/arXiv.2304.11062

    arXiv doi:10.48550/arxiv.2304.11062 2023

Show all 89 references
  1. [9]

    Llms are few-shot in-context low-resource language learners

    Cahyawijaya, S., Lovenia, H., and Fung, P. Llms are few-shot in-context low-resource language learners. arXiv preprint arXiv:2403.16512, 2024

    2024 arXiv

  2. [10]

    Importance of semantic representation: Dataless classification

    Chang, M.-W., Ratinov, L.-A., Roth, D., and Srikumar, V. Importance of semantic representation: Dataless classification. In Aaai, volume 2, pp.\ 830--835, 2008

    2008

  3. [11]

    Z., Wu, J., and Sun, J

    Chen, J., Yan, J., Chen, Q., Chen, D. Z., Wu, J., and Sun, J. Can a deep learning model be a sure bet for tabular prediction? In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp.\ 288–296, New York, NY, USA, 2024. Association fo...

    2024

  4. [12]

    and Guestrin, C

    Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp.\ 785--794, 2016

    2016

  5. [13]

    Notes from the ai frontier: Insights from hundreds of use cases

    Chui, M., Manyika, J., Miremadi, M., Henke, N., Chung, R., Nel, P., and Malhotra, S. Notes from the ai frontier: Insights from hundreds of use cases. McKinsey Global Institute, 2: 0 267, 2018

    2018

  6. [14]

    Support-vector networks

    Cortes, C. Support-vector networks. Machine Learning, 1995

    1995

  7. [15]

    and Hart, P

    Cover, T. and Hart, P. Nearest neighbor pattern classification. IEEE transactions on information theory, 13 0 (1): 0 21--27, 1967

    1967

  8. [16]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929, 2020. UR...

    2010 arXiv

  9. [17]

    Efficient and robust automated machine learning

    Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems 28 (2015), pp.\ 2962--2970, 2015

    2015

  10. [18]

    Auto-sklearn 2.0: Hands-free automl via meta-learning

    Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., and Hutter, F. Auto-sklearn 2.0: Hands-free automl via meta-learning. Journal of Machine Learning Research, 23: 0 1--61, 2022

    2022

  11. [19]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp.\ 1126--1135. PMLR, 2017

    2017

  12. [20]

    F., Feurer, M., and Bischl, B

    Fischer, S. F., Feurer, M., and Bischl, B. Openml-ctr23--a curated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop), 2023

    2023

  13. [21]

    and Witkowski, O

    Frans, K. and Witkowski, O. Population-based evolution optimizes a meta-learning objective. arXiv preprint arXiv:2103.06435, 2021

    2021 arXiv

  14. [22]

    C., and Schmidt, L

    Gardner, J., Perdomo, J. C., and Schmidt, L. Large scale transfer learning for tabular data via language modeling, 2024. URL https://arxiv.org/abs/2406.12031

    2024 arXiv

  15. [23]

    Meta-learning reduces the amount of data needed to build ai models in oncology

    Gevaert, O. Meta-learning reduces the amount of data needed to build ai models in oncology. British Journal of Cancer, 125 0 (3): 0 309--310, 2021

    2021

  16. [24]

    Generative adversarial nets

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural information processing systems, 27, 2014

    2014

  17. [25]

    Explaining and harnessing adversarial examples

    Goodfellow, I., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://arxiv.org/abs/1412.6572

    2015 arXiv

  18. [26]

    Revisiting deep learning models for tabular data

    Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34: 0 18932--18943, 2021

    2021

  19. [27]

    On embeddings for numerical features in tabular deep learning

    Gorishniy, Y., Rubachev, I., and Babenko, A. On embeddings for numerical features in tabular deep learning. In NeurIPS , 2022

    2022

  20. [28]

    Tabr: Tabular deep learning meets nearest neighbors

    Gorishniy, Y., Rubachev, I., Kartashev, N., Shlenskii, D., Kotelnikov, A., and Babenko, A. Tabr: Tabular deep learning meets nearest neighbors. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=rhgIgTSSxW

    2024

  21. [29]

    Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 0 507--520, 2022

    Grinsztajn, L., Oyallon, E., and Varoquaux, G. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 0 507--520, 2022

    2022

  22. [30]

    Tabllm: Few-shot classification of tabular data with large language models

    Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., and Sontag, D. Tabllm: Few-shot classification of tabular data with large language models. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelli...

    2023

  23. [31]

    Drift-resilient tab PFN : In-context learning distribution shifts on tabular data

    Helli, K., Schnurr, D., Hollmann, N., M \"u ller, S., and Hutter, F. Drift-resilient tab PFN : In-context learning distribution shifts on tabular data. In AutoML Conference 2024 (Workshop Track), 2024. URL https://openreview.net/forum?id=VbmqcoHpGT

    2024

  24. [32]

    Ho, T. K. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume 1, pp.\ 278--282. IEEE, 1995

    1995

  25. [33]

    Tabpfn: A transformer that solves small tabular classification problems in a second

    Hollmann, N., M \"u ller, S., Eggensperger, K., and Hutter, F. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848, 2022

    2022 arXiv

  26. [34]

    u ller, S., Purucker, L., Krishnakumar, A., K \

    Hollmann, N., M \"u ller, S., Purucker, L., Krishnakumar, A., K \"o rfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with a tabular foundation model. Nature, 637 0 (8045): 0 319--326, 2025

    2025

  27. [35]

    Multilayer feedforward networks are universal approximators

    Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neural networks, 2 0 (5): 0 359--366, 1989

    1989

  28. [36]

    Meta-learning in neural networks: A survey

    Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44 0 (9): 0 5149--5169, 2021

    2021

  29. [37]

    Tabtransformer: Tabular data modeling using contextual embeddings, 2020

    Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. Tabtransformer: Tabular data modeling using contextual embeddings, 2020. URL https://arxiv.org/abs/2012.06678

    2020 arXiv

  30. [38]

    N., and Plaat, A

    Huisman, M., Van Rijn, J. N., and Plaat, A. A survey of deep meta-learning. Artificial Intelligence Review, 54 0 (6): 0 4483--4541, 2021

    2021

  31. [39]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Ioffe, S. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015

    2015 arXiv

  32. [40]

    A survey on generative adversarial networks: Variants, applications, and training

    Jabbar, A., Li, X., and Omar, B. A survey on generative adversarial networks: Variants, applications, and training. ACM Computing Surveys (CSUR), 54 0 (8): 0 1--49, 2021

    2021

  33. [41]

    Categorical reparameterization with gumbel-softmax

    Jang, E., Gu, S., and Poole, B. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016

    2016 arXiv

  34. [42]

    Well-tuned simple nets excel on tabular datasets

    Kadra, A., Lindauer, M., Hutter, F., and Grabocka, J. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34: 0 23928--23941, 2021 a

    2021

  35. [43]

    Well-tuned simple nets excel on tabular datasets

    Kadra, A., Lindauer, M., Hutter, F., and Grabocka, J. Well-tuned simple nets excel on tabular datasets. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021 b . URL https://openreview.net/forum?id=d3k38LTDCyO

    2021

  36. [44]

    Lightgbm: A highly efficient gradient boosting decision tree

    Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017

    2017

  37. [45]

    Understanding catastrophic overfitting in single-step adversarial training

    Kim, H., Lee, W., and Lee, J. Understanding catastrophic overfitting in single-step adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 8119--8127, 2021

    2021

  38. [46]

    J., Grinsztajn, L., and Varoquaux, G

    Kim, M. J., Grinsztajn, L., and Varoquaux, G. CARTE : Pretraining and transfer for tabular learning. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=9kArQnKLDp

    2024

  39. [47]

    Tab DDPM : Modelling tabular data with diffusion models, 2023

    Kotelnikov, A., Baranchuk, D., Rubachev, I., and Babenko, A. Tab DDPM : Modelling tabular data with diffusion models, 2023. URL https://openreview.net/forum?id=EJka_dVXEcr

    2023

  40. [48]

    J., and Bengio, S

    Kurakin, A., Goodfellow, I. J., and Bengio, S. Adversarial machine learning at scale. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=BJm4T4Kgx

    2017

  41. [49]

    Zero-data learning of new tasks

    Larochelle, H., Erhan, D., and Bengio, Y. Zero-data learning of new tasks. In AAAI, volume 1, pp.\ 3, 2008

    2008

  42. [50]

    Metalearning: a survey of trends and technologies

    Lemke, C., Budka, M., and Gabrys, B. Metalearning: a survey of trends and technologies. Artificial intelligence review, 44: 0 117--130, 2015

    2015

  43. [51]

    B., Goldstein, T., Wilson, A

    Levin, R., Cherepanova, V., Schwarzschild, A., Bansal, A., Bruss, C. B., Goldstein, T., Wilson, A. G., and Goldblum, M. Transfer learning with deep tabular models. arXiv preprint arXiv:2206.15306, 2022

    2022 arXiv

  44. [52]

    Neural architecture optimization

    Luo, R., Tian, F., Qin, T., Chen, E., and Liu, T.-Y. Neural architecture optimization. Advances in neural information processing systems, 31, 2018

    2018

  45. [53]

    J., Mnih, A., and Teh, Y

    Maddison, C. J., Mnih, A., and Teh, Y. W. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016

    2016 arXiv

  46. [54]

    Towards deep learning models resistant to adversarial attacks

    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJzIBfZAb

    2018

  47. [55]

    Language models are few-shot learners

    Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 1, 2020

    2005 arXiv

  48. [56]

    When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36, 2024

    McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., and White, C. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36, 2024

    2024

  49. [57]

    P., Grabocka, J., and Hutter, F

    M \"u ller, S., Hollmann, N., Arango, S. P., Grabocka, J., and Hutter, F. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510, 2021

    2021 arXiv

  50. [58]

    Statistical foundations of prior-data fitted networks

    Nagler, T. Statistical foundations of prior-data fitted networks. In International Conference on Machine Learning, pp.\ 25660--25676. PMLR, 2023

    2023

  51. [59]

    STUNT : Few-shot tabular learning with self-generated tasks from unlabeled tables

    Nam, J., Tack, J., Lee, K., Lee, H., and Shin, J. STUNT : Few-shot tabular learning with self-generated tasks from unlabeled tables. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=_xlsjehDvlY

    2023

  52. [60]

    and Schulman, J

    Nichol, A. and Schulman, J. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2 0 (3): 0 4, 2018

    2018 arXiv

  53. [61]

    E., and Mitchell, T

    Palatucci, M., Pomerleau, D., Hinton, G. E., and Mitchell, T. M. Zero-shot learning with semantic output codes. Advances in neural information processing systems, 22, 2009

    2009

  54. [62]

    True few-shot learning with language models

    Perez, E., Kiela, D., and Cho, K. True few-shot learning with language models. Advances in neural information processing systems, 34: 0 11054--11070, 2021

    2021

  55. [63]

    Neural oblivious decision ensembles for deep learning on tabular data

    Popov, S., Morozov, S., and Babenko, A. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019

    1909 arXiv

  56. [64]

    A., and Lewis, M

    Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021

    2021 arXiv

  57. [65]

    V., and Gulin, A

    Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018

    2018

  58. [66]

    Qu, J., Holzm \"u ller, D., Varoquaux, G., and Morvan, M. L. Tabicl: A tabular foundation model for in-context learning on large data. arXiv preprint arXiv:2502.05564, 2025

    2025 arXiv

  59. [67]

    J., Burden, S

    Ratliff, L. J., Burden, S. A., and Sastry, S. S. On the characterization of local nash equilibria in continuous games. IEEE transactions on automatic control, 61 0 (8): 0 2301--2307, 2016

    2016

  60. [68]

    Revisiting pretraining objectives for tabular deep learning

    Rubachev, I., Alekberov, A., Gorishniy, Y., and Babenko, A. Revisiting pretraining objectives for tabular deep learning. arXiv preprint arXiv:2207.03208, 2022

    2022 arXiv

  61. [69]

    A., Xu, Z., Dickerson, J., Studer, C., Davis, L

    Shafahi, A., Najibi, M., Ghiasi, M. A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! Advances in neural information processing systems, 32, 2019

    2019

  62. [70]

    and Armon, A

    Shwartz-Ziv, R. and Armon, A. Tabular data: Deep learning is not all you need. Information Fusion, 81: 0 84--90, 2022

    2022

  63. [71]

    B., and Goldstein, T

    Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., and Goldstein, T. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021

    2021 arXiv

  64. [72]

    C., Thelin, S., and Klein, T

    Spinaci, M., Polewczyk, M., Hoffart, J., Kohler, M. C., Thelin, S., and Klein, T. PORTAL : Scalable tabular foundation models via content-specific tokenization. In NeurIPS 2024 Third Table Representation Learning Workshop, 2024. URL https://openreview.net/forum?id=TSZQvknbLO

    2024

  65. [73]

    Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021

    2021 arXiv

  66. [74]

    Regression shrinkage and selection via the lasso

    Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58 0 (1): 0 267--288, 1996

    1996

  67. [75]

    Tikhonov, A. N. Solution of incorrectly formulated problems and the regularization method. Sov Dok, 4: 0 1035--1038, 1963

    1963

  68. [76]

    L., Cabi, S., Eslami, S., Vinyals, O., and Hill, F

    Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S., Vinyals, O., and Hill, F. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34: 0 200--212, 2021

    2021

  69. [77]

    Meta-learning: A survey

    Vanschoren, J. Meta-learning: A survey. arXiv preprint arXiv:1810.03548, 2018

    2018 arXiv

  70. [78]

    K., Brahma, D., and Rai, P

    Verma, V. K., Brahma, D., and Rai, P. Meta-learning for generalized zero-shot learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 6062--6069, 2020

    2020

  71. [79]

    Wong, E., Rice, L., and Kolter, J. Z. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BJx040EFvH

    2020

  72. [80]

    N., Hutchins, D., and Szegedy, C

    Wu, Y., Rabe, M. N., Hutchins, D., and Szegedy, C. Memorizing transformers. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TrjbxzRcnf-

    2022

  73. [81]

    Zero-shot learning - the good, the bad and the ugly

    Xian, Y., Schiele, B., and Akata, Z. Zero-shot learning - the good, the bad and the ugly. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    2017

  74. [82]

    H., Schiele, B., and Akata, Z

    Xian, Y., Lampert, C. H., Schiele, B., and Akata, Z. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41 0 (9): 0 2251--2265, 2018

    2018

  75. [83]

    Making pre-trained language models great on tabular prediction

    Yan, J., Zheng, B., Xu, H., Zhu, Y., Chen, D., Sun, J., Wu, J., and Chen, J. Making pre-trained language models great on tabular prediction. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=anzIzGZuLi

    2024

  76. [84]

    Towards cross-table masked pretraining for web data mining

    Ye, C., Lu, G., Wang, H., Li, L., Wu, S., Chen, G., and Zhao, J. Towards cross-table masked pretraining for web data mining. In The Web Conference 2024, 2024 a . URL https://openreview.net/forum?id=9jj7cMOXQo

    2024

  77. [85]

    A closer look at deep learning on tabular data

    Ye, H.-J., Liu, S.-Y., Cai, H.-R., Zhou, Q.-L., and Zhan, D.-C. A closer look at deep learning on tabular data. CoRR, abs/2407.00956, 2024 b . URL https://doi.org/10.48550/arXiv.2407.00956

    2024 doi

  78. [86]

    You only propagate once: Accelerating adversarial training via maximal principle

    Zhang, D., Zhang, T., Lu, Y., Zhu, Z., and Dong, B. You only propagate once: Accelerating adversarial training via maximal principle. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processin...

    2019

  79. [87]

    Free adversarial training with layerwise heuristic learning

    Zhang, H., Shi, Y., Dong, B., Han, Y., Li, Y., and Kuang, X. Free adversarial training with layerwise heuristic learning. In International Conference on Image and Graphics, pp.\ 120--131. Springer, 2021

    2021

  80. [88]

    Xtab: Cross-table pretraining for tabular transformers

    Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., and Shoaran, M. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305.06090, 2023

    2023 arXiv

  81. [89]

    Varibad: Variational bayes-adaptive deep rl via meta-learning

    Zintgraf, L., Schulze, S., Lu, C., Feng, L., Igl, M., Shiarlis, K., Gal, Y., Hofmann, K., and Whiteson, S. Varibad: Variational bayes-adaptive deep rl via meta-learning. Journal of Machine Learning Research, 22 0 (289): 0 1--39, 2021. URL http://jmlr.org/papers/v22/21-0657.html

    2021

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.