Pith. sign in

REVIEW 30 cited by

SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.01342 v1 pith:ZWYL2XTT submitted 2021-06-02 cs.LG cs.AIstat.ML

Gowthami Somepalli , Micah Goldblum , Avi Schwarzschild , C. Bayan Bruss , Tom Goldstein This is my paper
classification cs.LGcs.AIstat.ML
keywords learningtabulardatadeepmethodmethodssaintattention
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even outperforms gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over a variety of benchmark tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 30 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

    cs.LG 2022-07 conditional novelty 8.0 of 10

    TabPFN is a Prior-Data Fitted Network that approximates Bayesian inference for small tabular classification by training a Transformer once on synthetic data drawn from a causal prior, then solves new tasks in a single...

  2. TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    TRL-Bench is a new multi-granular benchmark that releases 50 OpenML tables, linkage tasks, and a 47k-table data lake to show that tabular encoder performance is capability-specific rather than captured by one leaderboard.

  3. TabPFN-MT: A Natively Multitask In-Context Learner for Tabular Data

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    TabPFN-MT is a multitask in-context learner for tabular data that sets a new state-of-the-art on deep multitask learning for datasets under 1000 samples while reducing inference cost from O(T) to O(1) passes.

  4. Breaking the Curse with BAND: Nonparametric Distribution Estimation in High Dimensions

    stat.ML 2026-07 conditional novelty 6.0 of 10

    Sparse Bayesian-network factorization plus sparsity-aware regression yields polynomial TV rates for high-dimensional mixed-type distribution estimation, beating classical histogram rates under sparsity.

  5. RelPrism: A Multi-Faceted Pre-training Framework with Self-Generated Tasks for Relational Databases

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    RelPrism generates self-supervised pseudo-tasks from three attribute perspectives via multi-granularity clustering to improve representation learning for relational database prediction tasks.

  6. SAGA: A Sequence-Adaptive Generative Architecture for Multi-Horizon Probabilistic Forecasting with Adaptive Temporal Conformal Prediction

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    SAGA applies a decoder-only transformer with split conformal prediction to multi-horizon labor earnings forecasting on Swedish panel data, outperforming parametric baselines with guaranteed coverage intervals.

  7. Foundation Models for Credit Risk Prediction: A Game Changer?

    cs.LG 2026-05 conditional novelty 6.0 of 10

    Tabular foundation models, used zero-shot, match or beat tuned gradient boosting on average in credit PD and LGD benchmarks, with a larger edge on small datasets.

  8. Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    DistPFN is a test-time posterior adjustment that rescales TabPFN class probabilities to reduce overfitting to the training class distribution under label shift.

  9. Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    DistPFN is a test-time posterior adjustment technique that mitigates label shift in TabPFN by downweighting the training prior and emphasizing the model's predicted posterior, with a temperature-scaled variant, evalua...

  10. Weight-Informed Self-Explaining Clustering for Mixed-Type Tabular Data

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    WISE unifies representation via BEP, feature weighting via LOFO, two-stage clustering, and intrinsic explanations via DFI for mixed-type tabular data, outperforming baselines on six datasets.

  11. From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    Spline encodings for numerical features show task-dependent performance in tabular deep learning, with piecewise-linear encoding robust for classification and variable results for regression depending on spline family...

  12. Posterior-Calibrated Causal Circuits in Variational Autoencoders: Why Image-Domain Interpretability Fails on Tabular Data

    cs.LG 2026-03 unverdicted novelty 6.0 of 10

    Tabular VAEs show ~50% lower causal circuit modularity than image VAEs, with beta-VAE CES collapsing to 0.043 versus 0.133 due to reconstruction degradation, challenging direct transfer of image interpretability techniques.

  13. FEAT: A Linear-Complexity Foundation Model for Extremely Large Structured Data

    cs.LG 2026-03 unverdicted novelty 6.0 of 10

    FEAT is a linear-complexity structured data foundation model using dual-axis encoding, AFBM state-space models, and Conv-GLA to achieve O(N) scaling and permutation invariance while outperforming prior SFMs on real-wo...

  14. MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learning

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    MultiModalPFN extends TabPFN with modality projectors, a multi-head gated MLP, and cross-attention pooler to unify tabular and non-tabular inputs, outperforming prior methods on medical and general multimodal datasets.

  15. No Need to Train Your RDB Foundation Model

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Column-wise, parameter-free JUICE encodings let single-table ICL models solve multi-table RDB prediction tasks with no training or fine-tuning.

  16. LakeMLB: Data Lake Machine Learning Benchmark

    cs.LG 2026-02 conditional novelty 6.0 of 10

    LakeMLB is a new six-dataset benchmark for multi-table machine learning in data lakes; experiments find pretraining helps in Union scenarios and feature augmentation helps in Join scenarios.

  17. Pattern-Aware Graph Neural Networks for Handling Missing Data

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Explicitly encoding missingness patterns in bipartite GNNs yields ~17% average balanced-accuracy gains over GRAPE on seven UCI datasets with natural missingness, with random embeddings nearly matching learned ones.

  18. Modeling Decisions in Blockchain Analytics: A Leakage-Aware Evaluation of Tree-Based vs. Sequential Models

    cs.LG 2026-07 reject novelty 5.0 of 10

    The paper reports that XGBoost beats Transformer and BiLSTM models for Ethereum actor classification after masking certain high-signal contracts, and that sequence order adds little signal.

  19. Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models

    cs.LG 2026-07 conditional novelty 5.0 of 10

    All nine tested tabular foundation models degrade under distribution shift, and real-world pretraining provides no robustness advantage over synthetic pretraining.

  20. TabLoRA: Parameter-Efficient Low-Rank Ensemble Learning for Large-Scale Tabular Data

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Shared-backbone low-rank ensemble adapters let neural tabular models match much of full-ensemble accuracy on large data without linear parameter growth or frequent OOMs.

  21. RowNet: A Memory Transformer for Tabular Regression

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    RowNet uses a memory bank of labeled properties, two retrieval layers with attention, and a mixture-of-experts module to predict real estate price per square meter.

  22. Foundation Models for Credit Risk Prediction: A Game Changer?

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Tabular foundation models outperform standard methods in credit risk PD and LGD tasks, with larger gains on smaller datasets when used out-of-the-box.

  23. Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks

    cs.DB 2026-05 unverdicted novelty 5.0 of 10

    A BART-GraphSAGE hybrid achieves ROC-AUC 67.40 on one RelBench task, competitive with LightGBM but still behind specialized relational deep learning and foundation models.

  24. PRAGMA: Revolut Foundation Model

    cs.LG 2026-04 unverdicted novelty 5.0 of 10

    PRAGMA pre-trains a Transformer on heterogeneous banking events with a tailored self-supervised masked objective, yielding embeddings that support strong downstream performance on credit scoring, fraud detection, and ...

  25. PECKER: A Precisely Efficient Critical Knowledge Erasure Recipe For Machine Unlearning in Diffusion Models

    cs.AI 2026-04 conditional novelty 5.0 of 10

    Spline numerical encodings can match or beat standard scaling on tabular nets, but PLE is most robust for classification and learnable knots add substantial training cost.

  26. PECKER: A Precisely Efficient Critical Knowledge Erasure Recipe For Machine Unlearning in Diffusion Models

    cs.AI 2026-04 unverdicted novelty 5.0 of 10

    PECKER uses a saliency mask to prioritize parameter updates in distillation-based unlearning, achieving shorter training times for class and concept forgetting on CIFAR-10 and STL-10 while matching prior methods' efficacy.

  27. Is the Statistical Advantage Worth the Cost? An Empirical Comparison of KANs and MLPs for Structured Data Classification

    cs.LG 2026-07 reject novelty 4.0 of 10

    Across 12 fixed-hyperparameter tabular benchmarks, KANs beat MLPs on 9/12 datasets in accuracy and 10/12 in F1, yet cost ~16x parameters; the paper's significance tests are internally inconsistent.

  28. The Case for Model Science: Verify, Explore, Steer, Refine

    cs.AI 2026-05 unverdicted novelty 4.0 of 10

    Position paper proposing Model Science as a discipline to systematically analyze AI model behavior beyond benchmarks, drawing analogies from cognitive science, neuroscience, medicine, and agriculture.

  29. Tabular Data with Class Imbalance: Predicting Electric Vehicle Crash Severity with Pretrained Transformers (TabPFN) and Mamba-Based Models

    cs.LG 2025-09 unverdicted novelty 4.0 of 10

    Benchmarks TabPFN, MambaNet and MambaAttention on imbalanced EV crash severity classification with SMOTEENN resampling on Texas data, identifying intersection relation and speed limit as top features and MambaAttentio...

  30. Integrating SAINT with Tree-Based Models: A Case Study in Employee Attrition Prediction

    cs.LG 2026-04 unverdicted novelty 2.0 of 10

    Standalone tree-based models outperform both SAINT and SAINT-embedding hybrids for employee attrition prediction on tabular HR data.

Pith tools