Pith. sign in

REVIEW 4 major objections 5 minor 44 references

DOFEN: Deep Oblivious Forest ENsemble

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that DOFEN, a deep network built from randomly assembled relaxed oblivious decision trees, achieves state-of-the-art results among DNNs on the Tabular Benchmark and competes with gradient-boosted trees.

desk verdict A genuinely new tabular DNN architecture with a plausible but not airtight SOTA claim; worth serious review, but the headline margin over Trompt needs repeated-seed and budget-controlled evidence. read the letter →

arxiv 2412.16534 v2 pith:UJXYLOA3 submitted 2024-12-21 cs.LG stat.ML

Kuan-Yu Chen , Ping-Han Chiang , Hsin-Rung Chou , Chih-Sheng Chen , Tien-Hao Chang This is my paper · ORCID
classification cs.LGstat.ML
keywords tabulardatadeeplearningobliviousdecisiontreesensemblemethodsgradientboostingBenchmarkinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a deep network can stop lagging behind gradient-boosted decision trees on ordinary tabular data by borrowing the oblivious decision tree's structure. It introduces DOFEN, which generates soft per-column conditions, randomly combines them into relaxed oblivious decision trees (rODTs), groups rODTs into forests, and bags the forests' predictions. On the 73-dataset Tabular Benchmark the authors report that DOFEN ranks first among the DNN models and lands alongside CatBoost and XGBoost, especially on numerical-feature classification and regression tasks. If true, this matters because tabular data is the dominant format in finance, healthcare, and government, where tree ensembles have been the default choice.

What carries the argument

The central object is the relaxed oblivious decision tree (rODT), a decision table whose conditions are soft scores output by a per-column neural sub-network and whose feature/threshold combinations are fixed by a random permutation instead of learned splits. A pool of rODTs is reshaped from the condition matrix; forests are formed by sampling rODT pairs and taking a softmax-weighted sum of paired embedding vectors; and the final prediction is the bagged average over many forests. This machinery replaces the non-differentiable tree-search with random condition combination and differentiable weighting, and the bagging over forests is what the ablation credits for most of the gain.

What would settle it

A re-run of the Tabular Benchmark under the paper's protocol that finds DOFEN's average accuracy or R2 rank among DNNs below Trompt's, or that flips the medium-regression rankings once the missing large-task baselines are included, would settle against the central claim.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that the non-differentiable tree-building step can be replaced by random condition combination plus differentiable weighting without losing the inductive bias that makes oblivious trees strong on tabular data. DOFEN builds a condition matrix per sample, permutes and reshapes it into a pool of rODTs, learns an importance weight and embedding for each rODT, samples weighted subsets to form forests, and averages forest predictions. The authors report that this design yields the best average accuracy and R2 among all compared DNNs on the benchmark and places DOFEN in the top group with tree-based models, while also supplying feature-importance scores that match tree-based rankings.

Load-bearing premise

The load-bearing premise is that the benchmark comparison is fair: DOFEN and every baseline follow the same Tabular Benchmark splits and hyperparameter-search protocol, so the small reported margins reflect the architecture rather than extra tuning, larger compute budgets, or missing baselines.

Editorial extensions

If this is right

  • On numerical-feature tabular tasks, deep learning can now be reported as the top performer among DNNs and within the top three overall, so the default advice to use gradient-boosted trees loses its blanket force for such datasets.
  • Tree-inspired architectures that randomize structure rather than learn splits are a viable route to tabular DNNs, and the two-level forest ensemble is the component that prevents overfitting.
  • DOFEN's feature-importance scores match tree-based importance rankings on mushroom and wine datasets, so the model offers an interpretability story comparable to trees.
  • Because DOFEN needs no learned split thresholds, it can be trained end-to-end with standard optimizers, making the approach easy to port to new datasets.
  • The benchmark results suggest that DOFEN is a credible backbone for further tabular research, including self-supervised and multimodal settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the random-condition construction could be used as a frozen feature-extraction front end for semi-supervised or multimodal tabular learning, since the paper's conclusion suggests the rODT pool is a reusable representation rather than a task-specific tree.
  • Editorial extension: the pruning results imply a cheap testable improvement; keeping only the rODTs whose weights vary most across samples could cut DOFEN's long inference time without retraining.
  • Editorial extension: the paper's heterogeneous-feature gap suggests a concrete next experiment: applying condition-generation sub-networks with learned categorical thresholds rather than the current shared soft-condition mechanism might recover the missing performance on mixed-feature datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces DOFEN, a deep neural network architecture for tabular data that constructs relaxed oblivious decision trees (rODTs) by randomly combining per-column soft conditions, then ensembles rODTs into multiple forests with a bagging step. The method is evaluated on the Tabular Benchmark (73 datasets) with comparisons against tree-based and deep learning baselines, and the authors claim DOFEN achieves state-of-the-art results among DNNs and is competitive with gradient-boosted trees. The paper also includes an ablation study, an interpretability analysis, a stability analysis, and additional experiments on FT-Transformer and GRANDE benchmark datasets. Code is publicly released.

Significance. If the performance claim is robust, DOFEN is a meaningful contribution to deep learning for tabular data: it is a novel architecture that combines tree-inspired inductive biases with end-to-end neural training, and it reportedly outperforms prior DNNs on a widely used benchmark while remaining interpretable. The paper is thorough in its appendices, providing per-dataset results, computational efficiency analyses, and evaluations on additional benchmarks, and the released code supports reproducibility. However, the headline 'state-of-the-art among DNNs' claim is currently supported by a very small average margin over Trompt (0.0008 accuracy on medium classification), without repeated-seed intervals for the main tables, and the comparison may be affected by search-budget and baseline-coverage asymmetries. These issues bear directly on the central empirical claim.

major comments (4)
  1. [Section 4.2, Tables 31-33, Table 12] The claim that DOFEN is state-of-the-art among DNNs rests on an average accuracy margin of 0.0008 over Trompt (0.7805 vs 0.7797 in the optimal-hyperparameter medium classification results of Table 12). The main benchmark tables report a single value per model per dataset, with no confidence intervals or repeated-seed variation. Section 4.5 provides seed statistics only under default hyperparameters on four datasets where DOFEN ranks first; this does not characterize the searched configurations used in the headline comparisons. The authors should report mean and standard deviation (or confidence intervals) over multiple seeds for the main aggregate results, or at least for the comparison against Trompt, to establish that the 0.0008 margin is not within run-to-run noise.
  2. [Appendix I.2, Appendix C.3] The comparison may be unfair in terms of compute budget. Appendix I.2 gives DOFEN a search space that includes d up to 8, m up to 64, and Nhead up to 8, while Appendix C.3 reports that DOFEN's training time is approximately twice that of Trompt under optimal hyperparameters (1143.8 seconds vs 535.2 seconds on medium datasets). The paper states that it strictly follows the Tabular Benchmark protocols but does not specify the compute budget per model or per random-search iteration, nor the number of iterations used for each baseline. If all models were given the same number of random-search iterations, DOFEN consumed substantially more compute; if they were given equal wall-clock time, the number of iterations differs. The authors should state the exact protocol and, ideally, add an equal-compute or equal-iteration sensitivity analysis.
  3. [Appendix B.2, Tables 38-41] The large-task evaluation omits several baselines: MLP, HGBT, and RandomForest are missing from large classification and/or large regression results because they are absent from the official repository. Since the aggregate 'state-of-the-art among DNNs' claim covers all 73 datasets, the missing MLP baseline in particular means the claim is effectively decided by medium-sized tasks, where the margin over Trompt is very small. The authors should either run these baselines themselves (the code and hardware are available to them) or explicitly qualify the claim as covering only the tasks for which baselines are reported.
  4. [Table 1 (ablation)] The regression ablation relies on a post-hoc outlier removal. The cell for 'no forest ensemble' on all regression datasets is 0.3238, marked with an asterisk, and the footnote states that after removing the outlier dataset the average becomes 0.5457 (and 0.6107 for numerical-only regression). This is not a pre-specified outlier criterion. Moreover, the 'remove both' row reports 0.5441, which is higher than the 'no forest ensemble' value of 0.3238; this counterintuitive pattern is unexplained and weakens the claim that the forest ensemble is the key contributor. The authors should either pre-specify an outlier handling rule, report per-dataset ablation results, or both, so readers can assess the robustness of the ablation conclusions.
minor comments (5)
  1. [Section 2] There is a typo: 'we will futher discuss' should be 'we will further discuss'.
  2. [Figure 3 caption] The word 'Baggging' in the caption should be 'Bagging'.
  3. [Figure 1 and Figure 4 captions] The sentence 'The result are averaged over various datasets' should be 'The results are averaged over various datasets'.
  4. [Table 22 (Appendix D)] In the regression medium row, the 'Triple' column reports 0.7731, which is higher than the default 0.6796, contradicting the text that 'an increase in num_layers generally results in poorer performance.' Please clarify whether this is a typo or a genuine exception.
  5. [Section 6 (Limitation)] The sentence 'the workaround strategy of differentiable sparse selection proposed in this study' appears to describe a technique not introduced anywhere in the paper; DOFEN does not use differentiable sparse selection. This sentence should be revised or removed.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity; the central benchmark claim is self-contained, with only a minor non-load-bearing self-citation of Trompt.

full rationale

The paper's central claim is an empirical benchmark result: DOFEN is evaluated on the external Tabular Benchmark with standardized splits, preprocessing, and metrics, and its performance is compared against external baselines and prior DNN architectures. The derivation of DOFEN's architecture (Section 3) does not define any quantity in terms of the target outcome, and no equation reduces a reported prediction to a fitted input. The only notable self-citation is the use of Trompt (ref [24]), by four of the same authors, as a main baseline and as the source of some optimal-hyperparameter tree-based baseline numbers (Tables 12 and 14). This self-citation is not load-bearing for the architecture's validity: DOFEN is not constructed from or fitted to Trompt, and the headline DNN comparison (DOFEN versus Trompt) is computed in this paper on the shared benchmark protocol. The concerns raised by the skeptic about seed variance, search-budget asymmetry, and missing large-task baselines are correctness and evidence-quality issues, not circularity: they do not show that a result is equivalent to its inputs by construction. Therefore the circularity score is low, reflecting one minor self-citation with otherwise independent empirical content.

Assumptions & free parameters 9 free parameters · 6 assumptions · 2 invented entities

The central claim is empirical, resting on benchmark protocol correctness, fair baselines, and the architecture's random-condition design. Hyperparameters m, d, Nhead, drop_rate, and Nforest are tuned or chosen; standard training settings are fixed. No physical entities are introduced beyond the model components rODT and forest embeddings, which are listed for completeness.

free parameters (9)
  • m = 16 (default); searched in {16, 32, 64}
    Controls the number of conditions per column and the rODT pool size. Selected on validation during random search, Appendix I.2.
  • d = 4 (default); searched in {3, 4, 6, 8}
    rODT depth, number of conditions grouped into one tree. Selected on validation, Appendix I.2.
  • Nhead = 1 (default); searched in {1, 4, 8}
    Number of heads in the multi-head weighting extension. Selected on validation, Appendix I.2.
  • drop_rate = 0.0 (default); searched in {0.0, 0.1, 0.2}
    Dropout rate for regularization. Selected on validation, Appendix I.2.
  • Nforest = 100 (default); varied in {1, 10, 20, 50, 100, 400}
    Number of rODT forests used in the bagging ensemble. Performance and variance improve with Nforest, Section 4.5.
  • Nhidden = 128
    Hidden embedding dimension for each rODT, fixed in the default configuration, Table 4.
  • learning_rate = 1e-3
    AdamW learning rate fixed across all experiments, Appendix A.1.
  • batch_size = 256
    Fixed training batch size across all experiments, Appendix A.1.
  • num_epochs = 500
    Fixed training epochs without scheduling or early stopping, Appendix A.1.
assumptions (6)
  • domain assumption The Tabular Benchmark official implementation provides valid dataset splits, preprocessing, and evaluation metrics.
    Section 4.1 states the paper strictly follows official protocols; incorrect benchmark handling would invalidate all comparisons.
  • domain assumption Baselines in the comparison receive fair and representative hyperparameter tuning.
    Section 4.1 and Appendix I.2 define search spaces, but DOFEN's search space is compact and other models may be under- or over-tuned; no compute budget equality is enforced.
  • ad hoc to paper Randomly permuting and grouping conditions once at construction, Equation (2), yields a sufficiently diverse rODT pool.
    This is the core design bet of DOFEN. It is validated empirically in Sections 4.3 and 4.5, not proven.
  • domain assumption Weighted embedding aggregation with softmax and bagging over forests is optimizable by gradient descent and improves generalization.
    Algorithm 1 uses standard softmax operations, loss averaging, and bagging; the paper provides empirical support but no formal guarantee.
  • standard math Oblivious decision tree definitions and representational properties from Kohavi [11] and Lou and Obukhov [12] are correct.
    Section 2 builds on these prior results to motivate the relaxed ODT construction.
  • domain assumption Selecting the best model on a validation split, as prescribed by the benchmark, is a reliable model selection procedure.
    Section 4.1 and the figures use 'best model on valid set' after up to 100 random search iterations; this is standard but can overfit the validation set.
invented entities (2)
  • Relaxed Oblivious Decision Tree (rODT)
    purpose: A row of randomly grouped soft conditions that plays the role of a decision tree within the network.
    Defined in Section 3.1 and Equation (2). It is a model construct with no external falsifiable prediction beyond internal benchmark performance.
  • rODT forest embedding
    purpose: A weighted sum of sampled rODT embeddings used to make a prediction for each forest.
    Defined in Algorithm 1, line 6. It is an internal model construct without independent evidence outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DOFEN: Deep Oblivious Forest ENsemble." pith.science (2026). https://pith.science/paper/UJXYLOA3

@misc{pith2026241216534,
  author       = {Pith},
  title        = {Pith review of: DOFEN: Deep Oblivious Forest ENsemble},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJXYLOA3}},
  note         = {Machine review of arXiv:2412.16534}
}
read the original abstract

Deep Neural Networks (DNNs) have revolutionized artificial intelligence, achieving impressive results on diverse data types, including images, videos, and texts. However, DNNs still lag behind Gradient Boosting Decision Trees (GBDT) on tabular data, a format extensively utilized across various domains. In this paper, we propose DOFEN, short for \textbf{D}eep \textbf{O}blivious \textbf{F}orest \textbf{EN}semble, a novel DNN architecture inspired by oblivious decision trees. DOFEN constructs relaxed oblivious decision trees (rODTs) by randomly combining conditions for each column and further enhances performance with a two-level rODT forest ensembling process. By employing this approach, DOFEN achieves state-of-the-art results among DNNs and further narrows the gap between DNNs and tree-based models on the well-recognized benchmark: Tabular Benchmark \citep{grinsztajn2022tree}, which includes 73 total datasets spanning a wide array of domains. The code of DOFEN is available at: \url{https://github.com/Sinopac-Digital-Technology-Division/DOFEN}.

Figures

Figures reproduced from arXiv: 2412.16534 by the authors.

Figure 1
Figure 1. Evaluation results on the Tabular Benchmark. The model names are sorted by their [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Condition Generation: For each column, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) Relaxed ODT Forest Construction: First, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Results on medium-sized classification and regression datasets. The result are averaged [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Detailed network layer composite for ∆1. Linear( , ) LayerNorm( ) Dropout(drop_rate) ReLU() Linear( , ) LayerNorm( ) Dropout(drop_rate) [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 7
Figure 7. Figure 7: Detailed network layer composite for ∆3 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Overfitting arises when not applying sampling in the forest ensemble, affecting both (a) [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: In the covertype dataset, Figure 9a shows that the average weights of true positives differ [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: In the compass dataset, the weights wi of rODT are sorted based on the standard deviation calculated across true positive (TP) and true negative (TN) samples in the testing data. Figure 10a shows that the weights of TP samples differ significantly from those of TN sam…
Figure 11
Figure 11. Figure 11: Results on large-sized classification datasets. Classification. In Figure 11a, DOFEN even surpasses CatBoost to become the top performer. In Figure 11b, DOFEN and CatBoost clearly outperforms other models. In contrast to DOFEN, other tabular DNN models like FT-Transfo…
Figure 12
Figure 12. Figure 12: Results on large-sized regression datasets. Regression. In Figure 12a, the leading models remain DOFEN, XGBoost, and CatBoost. DOFEN’s proficiency in handling numerical features, further enhanced by the increased data volume, enables it to secure the top position once…
Figure 13
Figure 13. Figure 13: Results on each medium-sized classification datasets with only numerical features. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Results on each medium-sized classification datasets with heterogeneous features. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: Results on each medium-sized regression datasets with numerical features. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_15.png]
Figure 16
Figure 16. Figure 16: Results on each medium-sized regression datasets with heterogeneous features. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_16.png]
Figure 17
Figure 17. Figure 17: Results on each large-sized classification datasets with only numerical features. covertype road−safety 1 10 100 1 10 100 0.75 0.76 0.77 0.78 0.79 0.80 0.80 0.85 0.90 Number of random search iterations Test accuracy of best model (on valid set) up to this iteration Ca…
Figure 18
Figure 18. Figure 18: Results on each large-sized classification datasets with heterogeneous features. diamonds nyc−taxi−green−dec−2016 year 1 10 100 1 10 100 1 10 100 0.15 0.20 0.25 0.30 0.35 0.0 0.2 0.4 0.6 0.825 0.850 0.875 0.900 0.925 0.950 Number of random search iterations Test R2 sc…
Figure 19
Figure 19. Figure 19: Results on each large-sized regression datasets with numerical features. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]
Figure 20
Figure 20. Figure 20: Results on each large-sized regression datasets with heterogeneous features [PITH_FULL_IMAGE:figures/full_fig_p033_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 34 canonical work pages

  1. [1]

    Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems 2022, 35, 507–520

    Grinsztajn, L.; Oyallon, E.; Varoquaux, G. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems 2022, 35, 507–520

    work page 2022

  2. [2]

    V .; Gulin, A

    Prokhorenkova, L.; Gusev, G.; V orobev, A.; Dorogush, A. V .; Gulin, A. CatBoost: unbiased boosting with categorical features. Advances in neural information processing systems 2018, 31

    work page 2018

  3. [3]

    Xgboost: A scalable tree boosting system

    Chen, T.; Guestrin, C. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016; pp 785–794

    work page 2016

  4. [4]

    Revisiting deep learning models for tabular data

    Gorishniy, Y .; Rubachev, I.; Khrulkov, V .; Babenko, A. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems 2021, 34, 18932–18943

    work page 2021

  5. [5]

    Tabular data: Deep learning is not all you need.Information Fusion 2022, 81, 84–90

    Shwartz-Ziv, R.; Armon, A. Tabular data: Deep learning is not all you need.Information Fusion 2022, 81, 84–90

    work page 2022

  6. [6]

    Deep neural networks and tabular data: A survey

    Borisov, V .; Leemann, T.; Seßler, K.; Haug, J.; Pawelczyk, M.; Kasneci, G. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems 2022,

    work page 2022

  7. [7]

    McElfresh, D.; Khandagale, S.; Valverde, J.; Ramakrishnan, G.; Goldblum, M.; White, C.; others When Do Neural Nets Outperform Boosted Trees on Tabular Data? arXiv preprint arXiv:2305.02997 2023,

    arXiv 2023

  8. [8]

    Random forests

    Breiman, L. Random forests. Machine learning 2001, 45, 5–32

    work page 2001

Show all 44 references
  1. [9]

    Extremely randomized trees.Machine learning 2006, 63, 3–42

    Geurts, P.; Ernst, D.; Wehenkel, L. Extremely randomized trees.Machine learning 2006, 63, 3–42

    2006

  2. [10]

    Lightgbm: A highly efficient gradient boosting decision tree

    Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; Liu, T.-Y . Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 2017, 30

    2017

  3. [11]

    Bottom-up induction of oblivious read-once decision graphs

    Kohavi, R. Bottom-up induction of oblivious read-once decision graphs. European Conference on Machine Learning. 1994; pp 154–169

    1994

  4. [12]

    Bdt: Gradient boosted decision tables for high accuracy and scoring efficiency

    Lou, Y .; Obukhov, M. Bdt: Gradient boosted decision tables for high accuracy and scoring efficiency. Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017; pp 1893–1901

    2017

  5. [13]

    Quinlan, J. R. Induction of decision trees. Machine learning 1986, 1, 81–106

    1986

  6. [14]

    Neural oblivious decision ensembles for deep learning on tabular data

    Popov, S.; Morozov, S.; Babenko, A. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312 2019,

    1909 arXiv

  7. [15]

    Friedman, J. H. Stochastic gradient boosting. Computational statistics & data analysis 2002, 38, 367–378

    2002

  8. [16]

    Pedregosa, F. et al. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 2011, 12, 2825–2830

    2011

  9. [17]

    Deep forest

    Zhou, Z.-H.; Feng, J. Deep forest. National science review 2019, 6, 74–86

    2019

  10. [18]

    Ö.; Pfister, T

    Arik, S. Ö.; Pfister, T. Tabnet: Attentive interpretable tabular learning. Proceedings of the AAAI conference on artificial intelligence. 2021; pp 6679–6687

    2021

  11. [19]

    GradTree: Learning axis-aligned deci- sion trees with gradient descent

    Marton, S.; Lüdtke, S.; Bartelt, C.; Stuckenschmidt, H. GradTree: Learning axis-aligned deci- sion trees with gradient descent. Proceedings of the AAAI Conference on Artificial Intelligence. 2024; pp 14323–14331

    2024

  12. [20]

    GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data

    Marton, S.; Lüdtke, S.; Bartelt, C.; Stuckenschmidt, H. GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data. The Twelfth International Conference on Learning Represen- tations. 2024. 11

    2024

  13. [21]

    Tabtransformer: Tabular data modeling using contextual embeddings

    Huang, X.; Khetan, A.; Cvitkovic, M.; Karnin, Z. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678 2020,

    2012 arXiv

  14. [22]

    B.; Goldstein, T

    Somepalli, G.; Goldblum, M.; Schwarzschild, A.; Bruss, C. B.; Goldstein, T. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342 2021,

    2021 arXiv

  15. [23]

    Tabpfn: A transformer that solves small tabular classification problems in a second

    Hollmann, N.; Müller, S.; Eggensperger, K.; Hutter, F. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848 2022,

    2022 arXiv

  16. [24]

    Trompt: Towards a Better Deep Neural Network for Tabular Data

    Chen, K.-Y .; Chiang, P.-H.; Chou, H.-R.; Chen, T.-W.; Chang, T.-H. Trompt: Towards a Better Deep Neural Network for Tabular Data. arXiv preprint arXiv:2305.18446 2023,

    2023 arXiv

  17. [25]

    N.; Kaiser, Ł.; Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Advances in neural information processing systems 2017, 30

    2017

  18. [26]

    P.; Grabocka, J.; Hutter, F

    Müller, S.; Hollmann, N.; Arango, S. P.; Grabocka, J.; Hutter, F. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510 2021,

    2021 arXiv

  19. [27]

    Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I.; others Improving language under- standing by generative pre-training. 2018,

    2018

  20. [28]

    Advances in neural information processing systems 2019, 32

    Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; others Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 2019, 32

    2019

  21. [29]

    Decoupled weight decay regularization

    Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017,

    2017 arXiv

  22. [30]

    https://github.com/facebookresearch/fvcore/

    fvcore library. https://github.com/facebookresearch/fvcore/

  23. [31]

    G.; van Rijn, J

    Bischl, B.; Casalicchio, G.; Feurer, M.; Gijsbers, P.; Hutter, F.; Lang, M.; Mantovani, R. G.; van Rijn, J. N.; Vanschoren, J. Openml benchmarking suites. arXiv preprint arXiv:1708.03731 2017,

    2017 arXiv

  24. [32]

    2019;https://www.kaggle.com/code/donkeys/ lgbm-with-hyperopt-tuning/notebook , [Online; accessed 5-January-2023]

    Averagemn LGBM with hyperopt tuning. 2019;https://www.kaggle.com/code/donkeys/ lgbm-with-hyperopt-tuning/notebook , [Online; accessed 5-January-2023]

    2019

  25. [33]

    Understanding LightGBM Parameters (and How to Tune Them)

    Bahmani, M. Understanding LightGBM Parameters (and How to Tune Them). 2022; https: //neptune.ai/blog/lightgbm-parameters-guide , [Online; accessed 5-January-2023]. 12 Appendix Table of Contents A More DOFEN Settings 14 A.1 Default Hyperparameters Settings for DOFEN . . . . . ....

    2022

  26. [34]

    For categorical columns in particular, we employ embedding layers

    ∆1 - Generate conditions for each column: ∆1 is designed to generate conditions for both numerical and categorical data columns, as detailed in Figure 5. For categorical columns in particular, we employ embedding layers. These layers are utilized to transform categorical featu...

  27. [35]

    The relevant structures and processes are illustrated in Figure 6 and Figure 7

    ∆2 and ∆3 - Derive weights and make predictions: The layers represented by ∆2 and ∆3 are responsible for generating weights based on the combination of conditions and making predictions, respectively. The relevant structures and processes are illustrated in Figure 6 and Figure 7

  28. [36]

    This parameter represents the number of distinct categories in a given categorical column

    Most parameters and their notations used here have been defined in the main paper and Appendix A.1, despite num_categories. This parameter represents the number of distinct categories in a given categorical column. A.3 Actual Nestimator for each Dataset The Nestimator is calcu...

    1984

  29. [37]

    First, in line 6 and 7 of Algorithm 2, the embedding vector should be reshaped into a matrix for an additional head dimension, while the hidden dimension is reduced for each head

  30. [38]

    Second, in line 9 to 15 of Algorithm 2, we iterate through different heads, each weight and embedding of corresponding head are weighted summed to form part of the embedding for an rODT forest

  31. [39]

    Train" refers to training performance, and “Test

    Lastly, in line 8, 16, and 18 of Algorithm 2, we concatenate these embeddings from each head to form a complete embedding of an rODT forest. Moreover, this multi-head weighting does not increase much of computational cost, as the embedding dimension is reduced of each head. wi...

  32. [40]

    First, due to the lack of computational resources and time for these large size datasets, we only report the result of 5 different seeds, instead of the original setting that averages the result across 15 seeds from FT-Transformer paper

  33. [41]

    deep learning model

    Second, for model comparison, aside from models included in FT-Transformer paper itself (i.e. FT-Transformer, Catboost, and XGBoost), we additionally include two state-of-the-art deep learning models, Trompt and GRANDE, to show the effectiveness of DOFEN. The performance of FT...

    1922

  34. [43]

    We choose to use an adaptive batch size for each dataset, which is set to min{256, 2⌊log2( N 10 )⌋}, where N represents dataset size

    DOFEN originally set batch size as 256. We choose to use an adaptive batch size for each dataset, which is set to min{256, 2⌊log2( N 10 )⌋}, where N represents dataset size. This adjustment is made mainly because using too large batch size will have negative effect when traini...

  35. [44]

    L" indicates large datasets (data size > 10000),

    We set Nhead to 4 for DOFEN. This is because there exist several large-sized datasets in this benchmark, this adjustment is made to better accommodate with large-sized datasets. This adjustment has also been made in Appendix H.1 for the same reason. Based on the experiment res...

  36. [1000]

    For experimental settings, we strictly follow the settings mentioned in GRANDE paper

    according to the definition of Tabular Benchmark. For experimental settings, we strictly follow the settings mentioned in GRANDE paper. The experiment result is provided in Table 43. The performance of GRANDE, XGBoost, CatBoost, and NODE for both searched and default hyperpara...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.