Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Seeding gradient-boosted trees with a transformer's logits, so the trees learn the residuals, lets tabular classifiers beat either model alone across most dataset sizes.

desk verdict A simple, plausible fusion of transformer scores and GBDTs; the experiments are broad but the small-sample gains may partly be validation-tuning artifacts, so the headline needs softening. read the letter →

arxiv 2502.02672 v2 pith:RBHENUCN submitted 2025-02-04 cs.CL cs.LG

Mayuka Jayawardhana , Renbo , Samuel Dooley , Valeriia Cherepanova , Andrew Gordon Wilson , Frank Hutter , Colin White , Tom Goldstein
show 1 more author
Micah Goldblum
This is my paper · ORCID
classification cs.CLcs.LG MSC 68T0568T07
keywords tabulardatagradient-boosteddecisiontreeslargelanguagemodelsTabPFNboostingfew-shotclassificationensemblemethodsresiduallearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that transformer models and gradient-boosted decision trees are not rivals to choose between, but components of one pipeline. It proposes replacing the constant starting point of a GBDT with the scaled logits of an LLM (LLM-Boost) or of TabPFN (PFN-Boost), so that the trees learn the residual error of the transformer. If true, practitioners gain the small-data, header-aware strength of transformers and the scalability of GBDTs at the same time, with no fine-tuning of the transformer. The headline finding is that PFN-Boost achieves the best average performance among all tested methods for all but very small dataset sizes, while LLM-Boost is strongest in the very small regime.

What carries the argument

The carrying mechanism is the replacement of the GBDT's constant initial prediction with scaled transformer logits: a single scaling parameter $s$ multiplies the transformer scores before they enter the ensemble, and every subsequent tree is fit to the residual of the combined prediction. Setting $s=0$ recovers the plain GBDT and sending $s$ large recovers the transformer, so tuning $s$ on a validation split interpolates between the two. For LLM-Boost the scores are extracted as the negative mean per-token cross-entropy of each verbalized class label; for PFN-Boost they are TabPFN's unnormalized output scores. Because the transformer scores are computed once and reused, the extra training cost beyond a standard GBDT pipeline is small.

What would settle it

A reader could take the same 16 datasets, fix a large test split, and run PFN-Boost twice: once with the scaling parameter tuned by 30 hyperparameter trials on a 10-row validation split and once with the scaling parameter fixed to zero. If the tuned version does not beat the zero version on most datasets, the reported small-sample advantage is an artifact of tuning on a tiny validation set.

Watch

Extended reading notes

Core claim

The paper's central claim is that a gradient-boosted decision tree can be seeded with a pretrained transformer's raw class scores, and that the tree ensemble then learns the residuals of those transformer predictions, giving a model that inherits both the transformer's prior and the tree's scalability. The authors instantiate this as LLM-Boost, which seeds XGBoost with the negative mean per-token cross-entropy scores of an instruction-tuned LLM prompted with few-shot serialized rows, and PFN-Boost, which seeds XGBoost with TabPFN's raw scores. On 16 classification datasets at train sizes from 10 to 500 plus full datasets, they report that both boosted methods match or outperform each standalone component at the extremes and outperform both in between, that LLM-Boost is best in the very small regime, and that PFN-Boost attains the best average performance among all tested methods for all but very small dataset sizes, without fine-tuning.

Load-bearing premise

The load-bearing assumption is that a scaling parameter chosen on a tiny validation split of 10 to 50 rows will still be the right scale for the held-out test set, rather than just fitting noise in those few validation rows.

Editorial extensions

If this is right

  • PFN-Boost gives the best average AUC among all methods tested for every sample size except the smallest, so practitioners with datasets too large for TabPFN alone can still use TabPFN's pretraining through a cheap GBDT wrapper.
  • LLM-Boost is strongest at 10-100 training samples, beating the LLM alone, XGBoost alone, selection, and stacking, and it outperforms TabLLM's fine-tuning baseline in most sample sizes without any fine-tuning.
  • The fusion is model-agnostic: swapping XGBoost for LightGBM or swapping the primary LLM for Flan-T5-XXL or Llama-3-8B-Instruct still yields gains over baselines, with the size of the gain tracking the strength of the transformer.
  • Column headers carry measurable value: shuffling the headers degrades LLM-Boost at small sample sizes, confirming that the LLM's semantic reading of headers, not just its pretraining, is part of what the trees inherit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct corollary the paper does not test: the same residual-seeding trick should extend to other base learners and to regression, since nothing in the mechanism is specific to classification or to trees.
  • The method implies that the value of column headers can be quantified and traded off; the ablation shows headers matter most at small sample sizes, suggesting header quality, rather than dataset size alone, should determine whether to use an LLM seed.
  • A testable extension is to make the scaling parameter a function of dataset size or validation performance, or to select it by repeated cross-validation, which may stabilize the very-small-sample regime where tuning $s$ on 10 rows is noisy.
  • The PFN-Boost result suggests TabPFN's prior is strong enough that even at 500-1000 training rows it beats a from-scratch GBDT after residual correction, raising the question of whether larger-context tabular transformers could push the crossover point to even bigger datasets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a simple fusion mechanism, LLM-Boost and PFN-Boost, in which the raw prediction scores of a transformer (an LLM or TabPFN) are scaled by a tunable parameter and used as the initial prediction of a gradient-boosted decision tree, which then learns residuals from the transformer scores. Experiments on 16 tabular classification datasets across sample sizes 10 to full dataset compare these methods against the standalone GBDT, standalone transformer, validation-based selection, stacking, TabLLM, and AutoGluon. The central claims are that LLM-Boost and PFN-Boost outperform both of their constituent models on intermediate dataset sizes, and that PFN-Boost achieves the best average performance among all tested methods for all but the very smallest sample sizes.

Significance. If the central claims hold, the proposed methods offer a lightweight, no-fine-tuning way to inject pretrained transformer priors and natural-language column-header information into scalable GBDT pipelines, which is practically relevant for small and medium tabular classification problems. The paper's strengths include its breadth (16 datasets, multiple sample sizes, multiple LLM/GBDT combinations), the release of code, an ablation of column-header shuffling, and a comparison with TabLLM and AutoGluon. However, the evidence as presented has important gaps: the scaling parameter is tuned on extremely small validation splits, the evaluation mixes AUC and accuracy metrics when constructing aggregate curves, and no significance tests or error bars for the TabPFN baseline are provided for the often small performance differences.

major comments (3)
  1. [Section 4.2, Table 3 (rows with train/val = 10/10)] The scaling parameter s is tuned with 30 Optuna trials on validation sets as small as 10 rows, and the search space includes s=0 (Appendix A). Because s=0 reproduces the standalone GBDT, the tuning procedure is guaranteed to match or improve validation performance by construction, but this does not guarantee test improvement. With 10 validation points, the selected s can overfit noise, so the reported test gains may partly reflect the specific validation-split realization rather than a genuine fusion benefit. The authors should report the distribution of selected s values, evaluate sensitivity to validation size (e.g., by varying the validation split or using repeated cross-fitting), and compare against a selection rule with equivalent capacity (e.g., picking s from the same search grid with the same number of trials) to support the claim that the gains come from the fusion mechanism rather than from overfitting the validation set.
  2. [Section 5.1, Section 5.2, Tables 3 and 4] Table 3 reports AUC for PFN-Boost, while Table 4 reports accuracy for LLM-Boost; nevertheless, Section 5 states that rank and z-score are calculated 'based on AUC' and Figures 3 and 4 plot 'Average AUC' for both methods. Mixing accuracy and AUC across methods and then averaging them as a single 'AUC' metric is statistically invalid and makes the aggregate comparisons misleading. The authors should either compute AUC for all experiments (including LLM-Boost) or clearly separate accuracy-based and AUC-based analyses and adjust the wording of the claims accordingly.
  3. [Section 5.1, Table 3] TabPFN results are reported without standard errors, while the other methods have ± errors, and no significance tests are provided for the often tiny differences between PFN-Boost and TabPFN or XGBoost (e.g., Abalone train=10: PFN-Boost 0.7119±0.0309 vs TabPFN 0.7109; Churn train=10: PFN-Boost 0.7122±0.0000 vs XGB 0.7122±0.0000). The claim that PFN-Boost 'achieves the best average performance among all methods we test for all but very small dataset sizes' requires either error bars for all methods and paired significance tests across seeds, or a clear statement that the observed differences are not statistically significant. Without this, the headline claim is not yet supported by the presented evidence.
minor comments (5)
  1. [Figure 2 caption] The caption reads 'An few-shot prompt'; it should read 'A few-shot prompt'.
  2. [Figure 14 caption] The caption uses 'negligent' where 'negligible' is meant: 'the seed values in LLM-Boost are negligent' should be 'negligible'.
  3. [Section 4.2] The sentence 'We use separate validation folds so that test data is new used for HPO trials' contains a typo; it should read 'test data is never used for HPO trials'.
  4. [Appendix B] The text 'sample HPO initilization' should be 'sample HPO initialization'.
  5. [Section 3.3] The notation pred(0,i) and pred(1,i) is not fully defined; in particular, it is unclear whether the index i is zero-based or one-based and how the constant C is chosen. A precise definition of the prediction interval would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the boost pipeline is evaluated on held-out test data after fitting the scaling parameter on validation; no claim reduces to its inputs.

full rationale

The paper's derivation chain is an empirical pipeline: extract transformer scores, scale them by a tunable parameter s, seed the GBDT with these scores to learn residuals, and evaluate on held-out test data. No claimed result is defined in terms of the quantity it purports to predict. The scaling parameter s is fit on a validation split, but the paper explicitly states, 'We use separate validation folds so that test data is new used for HPO trials' (Section 4.2), and the central comparisons are test AUC/accuracy after HPO. The observation that intermediate s values can beat both standalone models is an empirical finding (Appendix F), not a consequence of the method's definition. The only author-overlapping citation is TabPFN, which is used as a fixed pretrained component rather than as a justification for the fusion's validity; TabPFN is a published external model with its own benchmark results. The 'Selection' baseline controls for choosing the better of the two standalone models on validation, so the reported gains over Selection are not forced by construction. There is no fitted parameter renamed as a prediction, no load-bearing self-citation chain, and no uniqueness claim imported from the authors' own prior work. The skeptical concern about tiny validation splits is a statistical validity issue, not circularity. Therefore, the paper's central claim is self-contained and empirically testable against held-out data.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The only genuinely new fitted quantity is the scaling parameter s; the rest are standard hyperparameters and domain assumptions about transformer usefulness.

free parameters (2)
  • scaling parameter s = per dataset/model, tuned with Optuna for 30 trials; search {0, LogUniform[1e-4, 1e4]}
    Controls how much the GBDT relies on transformer scores; the paper shows intermediate values are needed for the boost, making it a load-bearing fitted quantity.
  • GBDT hyperparameters = per dataset, 100 Optuna trials over the search space in Appendix A
    Standard model selection for XGBoost and LightGBM; not a conceptual weakness, but part of the pipeline that affects the comparison.
assumptions (4)
  • domain assumption Gradient boosting algorithms can be initialized with a per-sample additive offset while retaining the residual-fitting objective.
    The method's equivalence claim in Section 3.2, 'replacing the first tree ... with the transformer', assumes the GBDT implementation supports this without changing the loss optimization.
  • domain assumption LLM logits derived from 3-shot prompts are informative enough to improve residual learning.
    The method's benefit depends on the transformer scores carrying signal; the paper's own ablations show Llama-3-8B's unreliable label outputs reduce the gain (Section 5.4).
  • domain assumption Column headers carry semantic meaning and are correctly serialized into prompts.
    Section 5.5 shows shuffling headers reduces, but does not eliminate, the benefit; datasets with meaningless headers require prompt engineering (Section 6).
  • domain assumption Validation performance is a reliable guide for tuning the scaling parameter at very small sample sizes.
    With train/val splits as small as 10 samples and 30 tuning trials, the validation estimate is noisy; the method assumes the tuned s transfers to the test set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes." pith.science (2026). https://pith.science/paper/RBHENUCN

@misc{pith2026250202672,
  author       = {Pith},
  title        = {Pith review of: Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RBHENUCN}},
  note         = {Machine review of arXiv:2502.02672}
}
read the original abstract

Large language models (LLMs) perform remarkably well on tabular datasets in zero- and few-shot settings, since they can extract meaning from natural language column headers that describe features and labels. Similarly, TabPFN, a recent non-LLM transformer pretrained on numerous tables for in-context learning, has demonstrated excellent performance for dataset sizes up to a thousand samples. In contrast, gradient-boosted decision trees (GBDTs) are typically trained from scratch on each dataset without benefiting from pretraining data and must learn the relationships between columns from their entries alone since they lack natural language understanding. LLMs and TabPFN excel on small tabular datasets where a strong prior is essential, yet they are not competitive with GBDTs on medium or large datasets, since their context lengths are limited. In this paper, we propose a simple and lightweight approach for fusing large language models and TabPFN with gradient-boosted decision trees, which allows scalable GBDTs to benefit from the natural language capabilities and pretraining of transformers. We name our fusion methods LLM-Boost and PFN-Boost, respectively. While matching or surpassing the performance of the transformer at sufficiently small dataset sizes and GBDTs at sufficiently large sizes, LLM-Boost and PFN-Boost outperform both standalone components on a wide range of dataset sizes in between. We demonstrate state-of-the-art performance against numerous baselines and ensembling algorithms. We find that PFN-Boost achieves the best average performance among all methods we test for all but very small dataset sizes. We release our code at http://github.com/MayukaJ/LLM-Boost .

Figures

Figures reproduced from arXiv: 2502.02672 by the authors.

Figure 1
Figure 1. How LLM-Boost works for a toy cat vs. dog classification problem. Note that here the selected nodes are denoted in light blue. The scaling parameter denoted by S allows for controlling the effect of the LLM predictions on the tree ensemble. reduction. The exact loss extraction process will differ slightly depending on whether it is a Masked LLM or a Causal LLM. Finally, we center these raw scores around zero by subt… view at source ↗
Figure 2
Figure 2. An few-shot prompt for the UCI adult income dataset designed to extract the LLM prediction scores required for LLM-Boost. 3.3 The Scaling Parameter We use a scaling parameter s to scale the transformer scores before passing them on to the GBDT algorithm. By setting the scaling parameter to zero, our method is equivalent to the standalone GBDT; by making the scaling parameter very large, our method outputs prediction… view at source ↗
Figure 3
Figure 3. PFN-Boost, combining TabPFN and XGBoost outperforms ensemble baselines and standalone models across dataset sizes. Left: Average Z-score based on AUC performance across dataset sizes for PFN-Boost and other ensemble baselines. Right: Average AUC across dataset sizes. Further, in our experiments comparing LLM-Boost and PFN-Boost with TabLLM [18] and AutoGluon [11], we find our methods perform better in a majority of … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: LLM-Boost, combining Qwen-2.5-72B-Instruct and XGBoost, outperforms ensemble baselines and the standalone constituent models across small dataset sizes. Left: Average z-score based on AUC performance across dataset sizes for LLM-Boost and other ensemble baselines. Righ…
Figure 5
Figure 5. Figure 5: PFN-Boost outperforms LLM-Boost on larger datasets. Direct comparison of LLM-Boost with XGB+Qwen-2.5-72B-Instruct and PFN-Boost with XGB+TabPFN. We observe from this comparison that boosted TabPFN results are better except for on small dataset sizes. This is as expecte…
Figure 6
Figure 6. Figure 6: LLM-Boost benefits from meaningful column headers. To emulate datasets where column headers contain little or no semantic meaning we shuffle the column headers of the Adult Income Dataset and run LLM-Boost. We see that even with the shuffled column headers, the LLM doe…
Figure 7
Figure 7. Figure 7: Evaluations of XGBoost + TabPFN using metrics based on average AUC. [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Evaluations of XGBoost + Qwen-2.5-72B-Instruct using metrics based on average AUC. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Evaluations of XGBoost + Flan-T5-XXL using metrics based on average AUC. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Evaluations of XGBoost + Llama-8B-Instruct using metrics based on average AUC. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Evaluations of LightGBM + Flan-T5-XXL using metrics based on average AUC. [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: AUC comparison between TabLLM and LLM-Boost (Flan-T5-XXL+XGBoost) over different dataset sizes. LLM-Boost performs better than TabLLM in a majority of the dataset sizes without any finetuning. Performance of TabLLM was taken from the original work. LLM-Boost experimen…
Figure 13
Figure 13. Figure 13: LLM-Boost and PFN-Boost outperforms AutoGluon. These figures illustrate performance metrics based on average AUC calculated between XGBoost, AutoGluon, PFN￾Boost(TabPFN+XGBoost) and LLM-Boost(Flan-T5-XXL+XGBoost). All metrics were calculated over the mean of 5 seeds. …
Figure 14
Figure 14. Figure 14: LLM-Boost with intermediate scaling parameter values may lead to better performance than either standalone model. When the scaling parameter is close to 0, the performance of XGBoost + Flan-T5-XXL approaches that of XGBoost alone, since the seed values in LLM-Boost ar…
Figure 15
Figure 15. Figure 15: LLM-Boost performs better with larger LLMs and more in-context examples. The figure on the left illustrates the change in LLM-Boost performance with model size. The figure on the right showcases the change in LLM-Boost performance with varying number of few-shot examp…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FUTURE: Flexible Unlearning for Tree Ensemble

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FUTURE forgets training samples from tree ensembles by optimizing sigmoid-smoothed split thresholds and copying them back to the original discrete trees.

Reference graph

Works this paper leans on

30 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llama 3 model card, 2024

    AI@Meta. Llama 3 model card, 2024. URLhttps://github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md

    work page 2024

  2. [2]

    Optuna: A next-generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019

    2019

  3. [3]

    Tabnet: Attentive interpretable tabular learning.CoRR, abs/1908.07442, 2019

    Sercan Ömer Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning.CoRR, abs/1908.07442, 2019. URL http://arxiv.org/abs/1908.07442

    arXiv 1908

  4. [4]

    Deep neural networks and tabular data: A survey.IEEE Transactions on Neural Networks and Learning Systems, 2022

    Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey.IEEE Transactions on Neural Networks and Learning Systems, 2022

    2022

  5. [5]

    Hytrel: Hypergraph-enhanced tabular data representation learning

    Pei Chen, Soumajyoti Sarkar, Leonard Lausen, Balasubramaniam Srinivasan, Sheng Zha, Ruihong Huang, and George Karypis. Hytrel: Hypergraph-enhanced tabular data representation learning. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 32173–32193. Curran Ass...

    work page 2023

  6. [6]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016

    2016

  7. [7]

    Large language models are few (1)-shot table reasoners

    Wenhu Chen. Large language models are few (1)-shot table reasoners. InFindings of the Association for Computational Linguistics: EACL 2023, pages 1120–1130, 2023

    work page 2023

  8. [8]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean,...

    arXiv 2022

Show all 30 references
  1. [9]

    LIFT: Language-interfaced fine-tuning for non-language machine learning tasks

    Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. LIFT: Language-interfaced fine-tuning for non-language machine learning tasks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghy...

    2022

  2. [10]

    Uci machine learning repository, 2017

    Dheeru Dua and Casey Graff. Uci machine learning repository, 2017. URLhttp://archive. ics.uci.edu/ml

    2017

  3. [12]

    Autogluon-tabular: Robust and accurate automl for structured data.arXiv preprint arXiv:2003.06505, 2020

    Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data.arXiv preprint arXiv:2003.06505, 2020

    2003 arXiv

  4. [13]

    A decision-theoretic generalization of on-line learning and an application to boosting.Journal of computer and system sciences, 55(1):119–139, 1997

    Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting.Journal of computer and system sciences, 55(1):119–139, 1997

    1997

  5. [14]

    Friedman

    Jerome H. Friedman. Greedy function approximation: A gradient boosting machine.The Annals of Statistics, 29(5):1189 – 1232, 2001. doi: 10.1214/aos/1013203451. URL https: //doi.org/10.1214/aos/1013203451

    2001

  6. [15]

    Revisiting deep learning models for tabular data

    Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https: //openreview.net...

    2021

  7. [16]

    Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: //openreview.net/forum?id...

    2022

  8. [17]

    Large language models are zero-shot time series forecasters

    Nate Gruver, Marc Anton Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=md68e8iZK1

    2023

  9. [18]

    Tabllm: Few-shot classification of tabular data with large language models

    Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549–5581. PMLR, 2023

    2023

  10. [19]

    TabPFN: A transformer that solves small tabular classification problems in a second

    Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. InThe Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/ forum?id=cp5PvcI6w8_

    2023

  11. [20]

    Large language models for auto- mated data science: Introducing caafe for context-aware automated feature engineering

    Noah Hollmann, Samuel Müller, and Frank Hutter. Large language models for auto- mated data science: Introducing caafe for context-aware automated feature engineering. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Ad- vances in Neural Informati...

    2023

  12. [21]

    TABBIE: Pretrained repre- sentations of tabular data

    Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. TABBIE: Pretrained repre- sentations of tabular data. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editor...

    2021

  13. [22]

    Lightgbm: A highly efficient gradient boosting decision tree.Advances in neural information processing systems, 30, 2017

    Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree.Advances in neural information processing systems, 30, 2017

    2017

  14. [23]

    Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum

    Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer learning with deep tabular models. In The Eleventh International Conference on Learning Representations, 2023. URL https://open...

    2023

  15. [24]

    When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36, 2024

    Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakr- ishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36, 2024

    2024

  16. [25]

    Optimized feature generation for tabular data via llms with decision tree reasoning, 2024

    Jaehyun Nam, Kyuyoung Kim, Seunghyuk Oh, Jihoon Tack, Jaehyung Kim, and Jinwoo Shin. Optimized feature generation for tabular data via llms with decision tree reasoning, 2024. URL https://arxiv.org/abs/2406.08527

    2024 arXiv

  17. [26]

    Neural oblivious decision ensem- bles for deep learning on tabular data

    Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensem- bles for deep learning on tabular data. ArXiv, abs/1909.06312, 2019. URL https://api. semanticscholar.org/CorpusID:202573030

    1909 arXiv

  18. [27]

    Catboost: unbiased boosting with categorical features

    Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018

    2018

  19. [28]

    Tablet: Learning from instructions for tabular data.arXiv, 2023

    Dylan Slack and Sameer Singh. Tablet: Learning from instructions for tabular data.arXiv, 2023

    2023

  20. [29]

    Saint: Improved neural networks for tabular data via row attention and contrastive pre-training

    Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021

    2021 arXiv

  21. [30]

    Tap4llm: Table provider on sampling, augmenting, and packing semi-structured data for large language model reasoning.CoRR, abs/2312.09039, 2023

    Yuan Sui, Jiaru Zou, Mengyu Zhou, Xinyi He, Lun Du, Shi Han, and Dongmei Zhang. Tap4llm: Table provider on sampling, augmenting, and packing semi-structured data for large language model reasoning.CoRR, abs/2312.09039, 2023. URLhttps://doi.org/10.48550/arXiv.2312. 09039

    2023 doi arXiv

  22. [31]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLhttps://qwenlm. github.io/blog/qwen2.5/. 15 A Hyperparameter search spaces Table 2: Hyperparameter search spaces for our XGBoost and LightGBM experiments. XGBoost Parameter Distribution Max depth UniformInt[3...

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.